Using ARToolkit 5.4 with Unity

[chuen1118]

My project was originally built with ARToolkit6 + Unity 5.6.5.

However the camera calibration server was down, so I tried to upgrade to v5.4 to fix that. And since there is no v5.4 for Unity yet, I got the libARWrapper.a from ARToolkit 5.4 for iOS.

After some work, the project works fine in the editor, however when run on iPad, nothing can be detected.
When I looked into the log, the NFT pages were loaded successfully, but the projection matrix returns an array of NaN, and I suspect this is what caused the issue.

ARController (native): [info]Starting NFT tracking thread.
[info] Start tracking thread.
ARController: UpdateAR: ARToolKit video is running. Configuring Unity scene for video.
System.Xml.MonoFIXAttribute:.ctor(String)
UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
UnityEngine.Logger:Log(LogType, Object)
UnityEngine.Logger:Log(LogType, Object)
System.Collections.ObjectModel.Collection`1:InsertItem(Int32, T)
UnityEngine.Debug:Log(Object)
ARController:Log(String)
ARController:UpdateAR()
ARController:Update()
System.Xml.MonoFIXAttribute:.ctor(String)
UnityEngineInternal.WebRequestUtils:.cctor()
UnityEngineInternal.WebRequestUtils:.cctor()
UnityEngineInternal.WebRequestUtils:.cctor()
System.Security.Cryptography.AesTransform:Decrypt128(Byte[], Byte[], UInt32[])
 
(Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/DebugBindings.gen.cpp Line: 51)

ARController: Video 480x360@1Bpp (AR_PIXEL_FORMAT_420f)
System.Xml.MonoFIXAttribute:.ctor(String)
UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
UnityEngine.Logger:Log(LogType, Object)
UnityEngine.Logger:Log(LogType, Object)
System.Collections.ObjectModel.Collection`1:InsertItem(Int32, T)
UnityEngine.Debug:Log(Object)
ARController:Log(String)
ARController:UpdateAR()
ARController:Update()
System.Xml.MonoFIXAttribute:.ctor(String)
UnityEngineInternal.WebRequestUtils:.cctor()
UnityEngineInternal.WebRequestUtils:.cctor()
UnityEngineInternal.WebRequestUtils:.cctor()
System.Security.Cryptography.AesTransform:Decrypt128(Byte[], Byte[], UInt32[])
 
(Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/DebugBindings.gen.cpp Line: 51)

ARController: Projection matrix: [
NaN	NaN	NaN	NaN
NaN	NaN	NaN	NaN
NaN	NaN	NaN	NaN
NaN	NaN	NaN	NaN]
System.Xml.MonoFIXAttribute:.ctor(String)
UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
UnityEngine.Logger:Log(LogType, Object)
UnityEngine.Logger:Log(LogType, Object)
System.Collections.ObjectModel.Collection`1:InsertItem(Int32, T)
UnityEngine.Debug:Log(Object)
ARController:Log(String)
ARController:UpdateAR()
ARController:Update()
System.Xml.MonoFIXAttribute:.ctor(String)
UnityEngineInternal.WebRequestUtils:.cctor()
UnityEngineInternal.WebRequestUtils:.cctor()
UnityEngineInternal.WebRequestUtils:.cctor()
System.Security.Cryptography.AesTransform:Decrypt128(Byte[], Byte[], UInt32[])
 
(Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/DebugBindings.gen.cpp Line: 51)

ARController: Video size 480x360 will use texture size 480x360.
System.Xml.MonoFIXAttribute:.ctor(String)
UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
UnityEngine.Logger:Log(LogType, Object)
UnityEngine.Logger:Log(LogType, Object)
System.Collections.ObjectModel.Collection`1:InsertItem(Int32, T)
UnityEngine.Debug:Log(Object)
ARController:Log(String)
ARController:CreateVideoBackgroundMesh(Int32, Int32, Int32, Int32, Color32[]&, Texture2D&, Material&)
ARController:UpdateAR()
ARController:Update()
System.Xml.MonoFIXAttribute:.ctor(String)
UnityEngineInternal.WebRequestUtils:.cctor()
UnityEngineInternal.WebRequestUtils:.cctor()
UnityEngineInternal.WebRequestUtils:.cctor()
System.Security.Cryptography.AesTransform:Decrypt128(Byte[], Byte[], UInt32[])
 
(Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/DebugBindings.gen.cpp Line: 51)

ARController: Scene configured for video.
System.Xml.MonoFIXAttribute:.ctor(String)
UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
UnityEngine.Logger:Log(LogType, Object)
UnityEngine.Logger:Log(LogType, Object)
System.Collections.ObjectModel.Collection`1:InsertItem(Int32, T)
UnityEngine.Debug:Log(Object)
ARController:Log(String)
ARController:UpdateAR()
ARController:Update()
System.Xml.MonoFIXAttribute:.ctor(String)
UnityEngineInternal.WebRequestUtils:.cctor()
UnityEngineInternal.WebRequestUtils:.cctor()
UnityEngineInternal.WebRequestUtils:.cctor()
System.Security.Cryptography.AesTransform:Decrypt128(Byte[], Byte[], UInt32[])
 
(Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/DebugBindings.gen.cpp Line: 51)

In Bulk_Assembly-CSharp_0.cpp

extern "C"  bool ARNativePluginStatic_arwGetProjectionMatrix_m3881505666 (Il2CppObject * __this /* static, unused */, SingleU5BU5D_t1444911251* ___matrix0, const MethodInfo* method)
{
	typedef int8_t (DEFAULT_CALL *PInvokeFunc) (float*);

	// Marshaling of parameter '___matrix0' to native representation
	float* ____matrix0_marshaled = NULL;
	if (___matrix0 != NULL)
	{
		int32_t ____matrix0_Length = (___matrix0)->max_length;
		____matrix0_marshaled = il2cpp_codegen_marshal_allocate_array<float>(____matrix0_Length);
		memset(____matrix0_marshaled, 0, ____matrix0_Length * sizeof(float));
	}

	// Native function invocation
	int8_t returnValue = reinterpret_cast<PInvokeFunc>(arwGetProjectionMatrix)(____matrix0_marshaled);

	// Marshaling of parameter '___matrix0' back from native representation
	if (____matrix0_marshaled != NULL)
	{
		int32_t ____matrix0_Length = (___matrix0)->max_length;
		for (int32_t i = 0; i < ____matrix0_Length; i++)
		{
			(___matrix0)->SetAtUnchecked(static_cast<il2cpp_array_size_t>(i), (____matrix0_marshaled)[i]);
		}
	}

	// Marshaling cleanup of parameter '___matrix0' native representation
	il2cpp_codegen_marshal_free(____matrix0_marshaled);
	____matrix0_marshaled = NULL;

	return static_cast<bool>(returnValue);
}

After the line

int8_t returnValue = reinterpret_cast(arwGetProjectionMatrix)(____matrix0_marshaled);

value of ____matrix0_marshaled became NaN
And unfortunately I don't know how to trace further.

Then I tried replacing with library from v5.3.2 for Unity, and it works.

I have no idea what is causing this. Could someone please help with this?
Is it okay to use libARWrapper.a for iOS directly in Unity? Or do I need to re-build one? If yes, how can I do so? Is there a tutorial for that? Or did I miss some other libraries needed to make it work?

[ThorstenBux]

Hi @chuen1118, first of all, I'd like to point out that there is a significant API change from ARToolKit6 to ARToolKit5, there is a tutorial pointing out the steps involved to merge from 5 to 6 (http://augmentmy.world/migratetoartoolkit6) maybe that helps in reverse too.

Regarding your question, if you only target iOS devices with your Unity3D app you can use the precompiled ios library from other versions given that the external API didn't change. However, as you say that you can run your project inside the Unity3D editor successfully I assume that you updated all API changes or there haven't been any.

From your log I can see that that projection matrix returns NAN, however, that seems to occur before there was even an open camera stream as the line ARController: Scene configured for video. appears later in the log file.

Also, I'm missing a line that I believe should be indicating that a trackable was recognized inside the video stream.

As of now, I'd ask you if you could wait a bit longer until there is an official release of arunityX.

[chuen1118]

Thanks for your reply.

I thought the line ARController: Scene configured for video. always comes after projection matrix was set up? At least that is how I see in the log in my editor test or the successful run with ARTK 5.3.2.

As for not recognizing a trackable, I had assumed that the NAN projection matrix caused problem with the calculation of recognizing process. If this is not the case, then I might need to look into other possibilities.

Also, I would be happy to try out the official release of ARUnityX. Do you have an estimated release date of that?
I tried to swap the library because we hoped to update the APP before mid May thus tried for a quick fix. In the long run though, upgrading to an officially released version is more ideal.