国产av日韩一区二区三区精品,成人性爱视频在线观看,国产,欧美,日韩,一区,www.成色av久久成人,2222eeee成人天堂

Home Technology peripherals AI Up owners have already started to play tricks. Tencent opens up 'AniPortrait' to let photos sing and speak.

Up owners have already started to play tricks. Tencent opens up 'AniPortrait' to let photos sing and speak.

Apr 07, 2024 am 09:01 AM
Tencent industry

AniPortrait models are open source and can be played freely.


##""A new productivity tool for Xiaopozhan Ghost Zone."

Recently, a new project released by Tencent Open Source received such evaluation on Twitter. This project is AniPortrait, which generates high-quality animated portraits based on audio and a reference image.

Without further ado, let’s take a look at the demo that may be warned by a lawyer’s letter:
Up owners have already started to play tricks. Tencent opens up AniPortrait to let photos sing and speak.
Animation images Can also speak easily:
Up owners have already started to play tricks. Tencent opens up AniPortrait to let photos sing and speak.
The project has already received widespread praise after just a few days of being launched: the number of GitHub stars has exceeded 2,800.

Up owners have already started to play tricks. Tencent opens up AniPortrait to let photos sing and speak.

Let’s take a look at the innovations of AniPortrait.

Up owners have already started to play tricks. Tencent opens up AniPortrait to let photos sing and speak.

  • Paper title: AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation
  • Paper address: https ://arxiv.org/pdf/2403.17694.pdf
  • Code address: https://github.com/Zejun-Yang/AniPortrait

AniPortrait

The AniPortrait framework newly proposed by Tencent contains two modules: Audio2Lmk and Lmk2Video.

Audio2Lmk is used to extract Landmark sequences that capture complex facial expressions and lip movements from the audio input. Lmk2Video uses this Landmark sequence to generate temporally stable and consistent high-quality portrait videos.

Figure 1 gives an overview of the AniPortrait framework.

Up owners have already started to play tricks. Tencent opens up AniPortrait to let photos sing and speak.

Audio2Lmk

#音聲クリップのシーケンスの場(chǎng)合、ここでの目標(biāo)は、対応する 3D 顔メッシュ シーケンスとジェスチャ シーケンスを予測(cè)することです。

チームは、事前トレーニングされた wav2vec を使用してオーディオ特徴を抽出しました。このモデルはよく一般化されており、リアルな顔のアニメーションを生成するために重要な、音聲の発音とイントネーションを正確に認(rèn)識(shí)できます。得られたロバストな音聲特徴を利用することで、2 つの fc 層からなる?yún)g純なアーキテクチャを使用して、それらを 3D 顔メッシュに効率的に変換できます。チームは、このシンプルで単純な設(shè)計(jì)により、精度が保証されるだけでなく、推論プロセスの効率も向上することに気づきました。

音聲をジェスチャーに変換するタスクでは、チームが使用するバックボーン ネットワークは依然として同じ wav2vec です。ただし、このネットワークの重みは、audio-to-mesh モジュールのネットワークとは異なります。これは、ジェスチャはオーディオのリズムとピッチに密接に関連しているのに対し、オーディオからグリッドへのタスクは別の焦點(diǎn) (発音とイントネーション) に焦點(diǎn)を當(dāng)てているためです。以前の狀態(tài)の影響を考慮するために、チームはトランスフォーマー デコーダーを使用してジェスチャ シーケンスをデコードしました。このプロセスでは、モジュールはクロスアテンション メカニズムを使用してオーディオ機(jī)能をデコーダに統(tǒng)合します。上記 2 つのモジュールの場(chǎng)合、トレーニングに使用される損失関數(shù)は単純な L1 損失です。

メッシュとポーズ シーケンスを取得した後、透視投影を使用してそれらを 2D 顔ランドマーク シーケンスに変換します。これらのランドマークは、次のステージの入力信號(hào)です。

Lmk2Video

參照ポートレートと顔のランドマーク シーケンスが與えられた場(chǎng)合、チームの提案Lmk2Video は、時(shí)間的に一貫したポートレート アニメーションを作成できます。アニメーション プロセスでは、參照イメージとの一貫した外観を維持しながら、モーションをランドマーク シーケンスに合わせます。チームが採(cǎi)用したアイデアは、ポートレート アニメーションを一連のポートレート フレームとして表現(xiàn)することです。

Lmk2Video のネットワーク構(gòu)造設(shè)計(jì)は、AnimateAnyone からインスピレーションを受けています。バックボーン ネットワークは SD1.5 で、マルチフレーム ノイズ入力を一連のビデオ フレームに効果的に変換する時(shí)間モーション モジュールが統(tǒng)合されています。

さらに、同じく SD1.5 構(gòu)造を使用する ReferenceNet も使用されており、その機(jī)能は、參照畫像の外観情報(bào)を抽出し、それを畫像に統(tǒng)合することです。バックボーンネットワークです。この戦略的な設(shè)計(jì)により、出力ビデオ全體にわたって Face ID の一貫性が保たれます。

AnimateAnyone とは異なり、これにより PoseGuider のデザインが複雑になります。オリジナルのバージョンはいくつかの畳み込み層を統(tǒng)合しただけで、ランドマーク機(jī)能がバックボーン ネットワークの入力層の潛在機(jī)能と融合されました。 Tencent チームは、この初歩的なデザインでは唇の複雑な動(dòng)きを捉えることができないことに気づきました。そこで、対応するスケールのランドマーク機(jī)能をバックボーン ネットワークのさまざまなモジュールに統(tǒng)合するという、ControlNet のマルチスケール戦略を採(cǎi)用しました。これらの改善にもかかわらず、最終モデルのパラメーターの數(shù)は依然として非常に少ないです。

チームは、追加の入力として參照畫像のランドマークを使用するという別の改善も導(dǎo)入しました。 PoseGuider のクロスアテンション モジュールは、各フレーム內(nèi)の參照ランドマークとターゲット ランドマーク間の相互作用を容易にします。このプロセスにより、顔のランドマークと外観の間の関係を理解できる追加の手がかりがネットワークに提供され、ポートレート アニメーションがより正確な動(dòng)きを生成するのに役立ちます。
実験

##実裝の詳細(xì)

Audio2Lmk ステージで使用されるバックボーン ネットワークは wav2vec2.0 です。 3D メッシュと 6D ポーズを抽出するために使用されるツールは MediaPipe です。 Audio2Mesh のトレーニング データは Tencent の內(nèi)部データセットから取得されており、これには 1 人の話者からの 1 時(shí)間近くの高品質(zhì)音聲データが含まれています。

MediaPipe によって抽出された 3D メッシュの安定性を確保するため、録畫中、出演者の頭の位置は安定し、カメラに面しています。 Audio2Pose のトレーニングでは HDTF を使用します。すべてのトレーニング操作は、Adam オプティマイザーを使用して単一の A100 で実行され、學(xué)習(xí)率は 1e-5 に設(shè)定されます。

Lmk2Video プロセスは 2 ステップのトレーニングを使用します。方法。

#最初のステップ フェーズでは、モーション モジュールに関係なく、バックボーン ネットワーク ReferenceNet と PoseGuider の 2D コンポーネントのトレーニングに焦點(diǎn)を當(dāng)てます。後続のステップでは、モーション モジュールのトレーニングに集中するために、他のすべてのコンポーネントがフリーズされます。モデルをトレーニングするために、ここでは 2 つの大規(guī)模で高品質(zhì)の顔ビデオ データセット、VFHQ と CelebV-HQ が使用されます。すべてのデータは MediaPipe を介して渡され、2D 顔ランドマークが抽出されます。唇の動(dòng)きに対するネットワークの感度を向上させるために、チームのアプローチは、2D ランドマークに基づいてポーズ畫像をレンダリングするときに、上唇と下唇に異なる色で注釈を付けることでした。

#すべての畫像は 512x512 に再スケールされました。モデルは 4 つの A100 GPU を使用してトレーニングされ、各ステップには 2 日かかりました。オプティマイザは AdamW で、學(xué)習(xí)率は 1e-5 に固定されています。

#実験結(jié)果

図 2 に示すように、新しいアニメーションによって得られたアニメーションは、この方法は品質(zhì)と現(xiàn)実性に優(yōu)れています。

Up owners have already started to play tricks. Tencent opens up AniPortrait to let photos sing and speak.

さらに、ユーザーは途中で 3D 表現(xiàn)を編集して、最終出力を変更することができます。たとえば、次のビデオに示されているように、ユーザーはソースからランドマークを抽出し、その ID 情報(bào)を変更して顔の再現(xiàn)を?qū)g現(xiàn)できます。
Up owners have already started to play tricks. Tencent opens up AniPortrait to let photos sing and speak.詳細(xì)については、元の論文を參照してください。

The above is the detailed content of Up owners have already started to play tricks. Tencent opens up 'AniPortrait' to let photos sing and speak.. For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undress AI Tool

Undress AI Tool

Undress images for free

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

DeepMind robot plays table tennis, and its forehand and backhand slip into the air, completely defeating human beginners DeepMind robot plays table tennis, and its forehand and backhand slip into the air, completely defeating human beginners Aug 09, 2024 pm 04:01 PM

But maybe he can’t defeat the old man in the park? The Paris Olympic Games are in full swing, and table tennis has attracted much attention. At the same time, robots have also made new breakthroughs in playing table tennis. Just now, DeepMind proposed the first learning robot agent that can reach the level of human amateur players in competitive table tennis. Paper address: https://arxiv.org/pdf/2408.03906 How good is the DeepMind robot at playing table tennis? Probably on par with human amateur players: both forehand and backhand: the opponent uses a variety of playing styles, and the robot can also withstand: receiving serves with different spins: However, the intensity of the game does not seem to be as intense as the old man in the park. For robots, table tennis

The first mechanical claw! Yuanluobao appeared at the 2024 World Robot Conference and released the first chess robot that can enter the home The first mechanical claw! Yuanluobao appeared at the 2024 World Robot Conference and released the first chess robot that can enter the home Aug 21, 2024 pm 07:33 PM

On August 21, the 2024 World Robot Conference was grandly held in Beijing. SenseTime's home robot brand "Yuanluobot SenseRobot" has unveiled its entire family of products, and recently released the Yuanluobot AI chess-playing robot - Chess Professional Edition (hereinafter referred to as "Yuanluobot SenseRobot"), becoming the world's first A chess robot for the home. As the third chess-playing robot product of Yuanluobo, the new Guoxiang robot has undergone a large number of special technical upgrades and innovations in AI and engineering machinery. For the first time, it has realized the ability to pick up three-dimensional chess pieces through mechanical claws on a home robot, and perform human-machine Functions such as chess playing, everyone playing chess, notation review, etc.

Claude has become lazy too! Netizen: Learn to give yourself a holiday Claude has become lazy too! Netizen: Learn to give yourself a holiday Sep 02, 2024 pm 01:56 PM

The start of school is about to begin, and it’s not just the students who are about to start the new semester who should take care of themselves, but also the large AI models. Some time ago, Reddit was filled with netizens complaining that Claude was getting lazy. "Its level has dropped a lot, it often pauses, and even the output becomes very short. In the first week of release, it could translate a full 4-page document at once, but now it can't even output half a page!" https:// www.reddit.com/r/ClaudeAI/comments/1by8rw8/something_just_feels_wrong_with_claude_in_the/ in a post titled "Totally disappointed with Claude", full of

Tencent 2025 campus recruitment starts: Graduation time expanded from one year to two years Tencent 2025 campus recruitment starts: Graduation time expanded from one year to two years Aug 07, 2024 pm 08:17 PM

According to news from this site on August 7, Tencent’s 2025 campus recruitment was officially launched yesterday. Following the recruitment of interns in 2024 and the “Qingyun Plan” AI large model recruitment special project, it once again issued an “enrollment expansion” signal: not only the recruitment scale is larger than the previous two years There has been a huge growth, and the graduation time range for people has also been further expanded. According to reports, Tencent’s campus recruitment in 2025 will open more than 70 positions in five major categories: technology, product, market, design, and function. There will be many changes in Tencent’s campus recruitment this year. The graduation time range for campus recruitment will be expanded from one year to two years. Students who graduate from January 2024 to December 2025 (Graduation Certificate shall prevail in Mainland China, Degree Certificate shall prevail in Hong Kong, Macao, Taiwan and overseas regions) can apply through Tencent’s official recruitment website and “Tencent

Li Feifei's team proposed ReKep to give robots spatial intelligence and integrate GPT-4o Li Feifei's team proposed ReKep to give robots spatial intelligence and integrate GPT-4o Sep 03, 2024 pm 05:18 PM

Deep integration of vision and robot learning. When two robot hands work together smoothly to fold clothes, pour tea, and pack shoes, coupled with the 1X humanoid robot NEO that has been making headlines recently, you may have a feeling: we seem to be entering the age of robots. In fact, these silky movements are the product of advanced robotic technology + exquisite frame design + multi-modal large models. We know that useful robots often require complex and exquisite interactions with the environment, and the environment can be represented as constraints in the spatial and temporal domains. For example, if you want a robot to pour tea, the robot first needs to grasp the handle of the teapot and keep it upright without spilling the tea, then move it smoothly until the mouth of the pot is aligned with the mouth of the cup, and then tilt the teapot at a certain angle. . this

Distributed Artificial Intelligence Conference DAI 2024 Call for Papers: Agent Day, Richard Sutton, the father of reinforcement learning, will attend! Yan Shuicheng, Sergey Levine and DeepMind scientists will give keynote speeches Distributed Artificial Intelligence Conference DAI 2024 Call for Papers: Agent Day, Richard Sutton, the father of reinforcement learning, will attend! Yan Shuicheng, Sergey Levine and DeepMind scientists will give keynote speeches Aug 22, 2024 pm 08:02 PM

Conference Introduction With the rapid development of science and technology, artificial intelligence has become an important force in promoting social progress. In this era, we are fortunate to witness and participate in the innovation and application of Distributed Artificial Intelligence (DAI). Distributed artificial intelligence is an important branch of the field of artificial intelligence, which has attracted more and more attention in recent years. Agents based on large language models (LLM) have suddenly emerged. By combining the powerful language understanding and generation capabilities of large models, they have shown great potential in natural language interaction, knowledge reasoning, task planning, etc. AIAgent is taking over the big language model and has become a hot topic in the current AI circle. Au

Hongmeng Smart Travel S9 and full-scenario new product launch conference, a number of blockbuster new products were released together Hongmeng Smart Travel S9 and full-scenario new product launch conference, a number of blockbuster new products were released together Aug 08, 2024 am 07:02 AM

This afternoon, Hongmeng Zhixing officially welcomed new brands and new cars. On August 6, Huawei held the Hongmeng Smart Xingxing S9 and Huawei full-scenario new product launch conference, bringing the panoramic smart flagship sedan Xiangjie S9, the new M7Pro and Huawei novaFlip, MatePad Pro 12.2 inches, the new MatePad Air, Huawei Bisheng With many new all-scenario smart products including the laser printer X1 series, FreeBuds6i, WATCHFIT3 and smart screen S5Pro, from smart travel, smart office to smart wear, Huawei continues to build a full-scenario smart ecosystem to bring consumers a smart experience of the Internet of Everything. Hongmeng Zhixing: In-depth empowerment to promote the upgrading of the smart car industry Huawei joins hands with Chinese automotive industry partners to provide

At the World Robot Conference, this domestic robot carrying 'the hope of future elderly care' was surrounded At the World Robot Conference, this domestic robot carrying 'the hope of future elderly care' was surrounded Aug 22, 2024 pm 10:35 PM

At the World Robot Conference being held in Beijing, the display of humanoid robots has become the absolute focus of the scene. At the Stardust Intelligent booth, the AI ??robot assistant S1 performed three major performances of dulcimer, martial arts, and calligraphy in one exhibition area, capable of both literary and martial arts. , attracted a large number of professional audiences and media. The elegant playing on the elastic strings allows the S1 to demonstrate fine operation and absolute control with speed, strength and precision. CCTV News conducted a special report on the imitation learning and intelligent control behind "Calligraphy". Company founder Lai Jie explained that behind the silky movements, the hardware side pursues the best force control and the most human-like body indicators (speed, load) etc.), but on the AI ??side, the real movement data of people is collected, allowing the robot to become stronger when it encounters a strong situation and learn to evolve quickly. And agile

See all articles