iFacialMocap communication specifications

iFacialMocap allows 3rd party tool developers to receive facial data using UDP or TCP / IP protocols without going through PC software.

・ To receive data via UDP

If you send the string "iFacialMocap_sahuasouryya9218sauhuiayeta91555dy3719" from the PC side to port 49983 on the iOS side, iFacialMocap will return a reply to port 49983 with UDP at 60 FPS.

・Python Example

from socket import *

DstIP = "192.168.1.7"

DstPort = 49983

DstAddr = (DstIP,DstPort)

udpClntSock = socket(AF_INET, SOCK_DGRAM)

data = "iFacialMocap_sahuasouryya9218sauhuiayeta91555dy3719"

data = data.encode('utf-8')

udpClntSock.sendto(data,DstAddr)

server = socket(AF_INET, SOCK_DGRAM)

server.bind(("", 49983))

server.settimeout(0.05)

while True:

　try:

　　messages, address = server.recvfrom(8192)

　　print(messages.decode('utf-8'))

　except:

　　pass

・The character string sent per frame is as follows.

・The character strings sent per frame are as follows.

BlendShape Name-Parameters (0 ~ 100) | BlendShape Names-Parameters (0 ~ 100) | .... | = head # Euler angles X (degree), Euler angles Y, Euler angles Z, Position values X, Position values Y, Position values Z | rightEye #Euler angles X, Euler angles Y, Euler angles Z | leftEye #Euler angles X, Euler angles Y, Euler angles Z |

Angle-related data is sent in degrees, not radians.

-From iOS ver1.1.8, an external command has been added to make it compatible with Facemotion 3d.

If you add the string "|sendDataVersion=v2" as shown below, the values of the BlendShape parameter sent will be separated by "&".

iFacialMocap_sahuasouryya9218sauhuiayeta91555dy3719|sendDataVersion=v2

[Example]

Normal

mouthSmile_R-0

sendDataVersion = v2

mouthSmile_R & 0

The reason for this process is that Facemotion 3d can send a negative value to the blendShape value.
This is because there was a possibility of a collision if "-" was used as the delimiter.

・ To receive data via TCP / IP

If it is UDP, frames may be dropped, so there is also a method of receiving with TCP / IP.

If you send the character string "iFacialMocap_UDPTCP_sahuasouryya9218sauhuiayeta91555dy3719" to port 49983 on the iOS side by UDP, the data will be sent to port 49986 on the PC side.

The character string to be sent is just the character string "___iFacialMocap" added to the character string sent at the time of UDP.

·sample

BlendShape Name-Parameter (0 ~ 100) | BlendShape Name-Parameter (0 ~ 100) | .... | = head # Euler angles X, Euler angles Y, Euler angles Z, moving values X, moving values Y, moving values Z | rightEye #Euler angles X, Euler angles Y, Euler angles Z | leftEye #Euler angles X, Euler angles Y, Euler angles Z | ___ iFacialMocap

When receiving from iOS to PC with TCP / IP, a character string with broken frames will be sent, so use "___iFacialMocap" at the end as a delimiter.

To stop sending TCP / IP, send the string "iFacialMocap_UDPTCPSTOP_sahuasouryya9218sauhuiayeta91555dy3719" to port 49983.

・Sample Python usage example received over TCP / IP

from socket import socket, AF_INET, SOCK_DGRAM, SOCK_STREAM

#UDP send

HOST = ''

PORT = 49983

ADDRESS = "192.168.1.7" #iOS address

s = socket(AF_INET, SOCK_DGRAM)

msg = "iFacialMocap_UDPTCP_sahuasouryya9218sauhuiayeta91555dy3719"

s.sendto(msg.encode(), (ADDRESS, PORT))

s.close()

#create tcp server

with socket(AF_INET, SOCK_STREAM) as tcp_server:

tcp_server.bind(('192.168.1.5', 49986)) #192.168.1.5 is pc address

tcp_server.listen(1)

while True:

conn, addr = tcp_server.accept()

with conn:

while True:

data = conn.recv(10000)

print(data.decode('utf-8'))

-From iOS ver1.1.8, an external command has been added to make it compatible with Facemotion 3d.

If you add the string "|sendDataVersion=v2" as shown below, the values of the BlendShape parameter sent will be separated by "&".

iFacialMocap_UDPTCP_sahuasouryya9218sauhuiayeta91555dy3719|sendDataVersion=v2

[Example]

Normal

mouthSmile_R-0

sendDataVersion = v2

mouthSmile_R & 0

The reason for this process is that Facemotion 3d can send a negative value to the blend shape value. This is because there was a possibility of a collision if "-" was used as the delimiter.

・ To receive recorded data in iOS

If you send the character string "iFacialMocap_bakeRecordedAnimation_start" to port 49983 on the iOS side by UDP, the recorded data will be sent to port 49987 on the PC side by TCP / IP.

The string that is sent is in the format [".....", ".....", "....."] ___iFacialMocap.

The [".....", ".....", "....."] part is in JSON format and stores the same character strings used in UDP and TCP / IP.

Data recorded at 60 FPS is always stored. (Even if the processing drops to 30FPS in iOS, it will be recorded at 60FPS)

This isn't what iFacialMocap was supposed to do when it was first released, but 3rd party developers are free to use it. However, the development of iFacialMocap will gradually shift to another successor app, and the successor app will be treated like a higher version of iFacialMocap. However, iFacialMocap itself will not be deprecated. If you are using this feature, please note that it is uncertain how it will be handled by the successor app that will be released in the future. (Specifically, the price will be divided for each function in detail.)

・Run the lookForward button

Send the string "iFacialMocap_lookForward" to iOS port 49983.

・Direct streaming to Unity

The following scripts can be used by attaching them to game objects.

It can stream with both iFacialMocap and Facemotion3d.

https://drive.google.com/file/d/10cnZRlglrHmZIv-ECT66eqNZNsiWhMtf/view?usp=sharing

It works by specifying the object name as shown in the following image.

This is a script intended for FBX files, not VRM.

If you wish to use this in VRM, please create your own or email me.