[Solved] Missing the metadata

For help and support with Universal Media Server
Forum rules
Please make sure you follow the Problem Reporting Guidelines before posting if you want a reply
User avatar
mik_s
Moderator
Posts: 1115
Joined: Wed Aug 23, 2017 11:03 pm
Location: UK

Re: Missing the metadata

Post by mik_s »

You are not seeing the JavaScript error on the page, you are looking at the source code and it is normal for it to be there.
If JavaScript was turned off you will get a white screen with just that message.

I think I can see what is happening now. Several things have been going on in your past logs so it did not make sense.

The first one there were errors about the database being closed and this would have been caused by the failed database update.

Code: Select all

TRACE 2023-02-07 17:57:44.114 [OpenSubtitles background worker 1] Database is closed
The second one was due to a problem with the API and that would have effected everyone

Code: Select all

DEBUG 2023-02-08 18:05:25.339 [OpenSubtitles background worker 1] Error with HttpURLConnection: {}
java.net.SocketTimeoutException: Read timed out ...
The third one wasn't much help either but it showed that it was trying to get metadata as it was full of this message

Code: Select all

TRACE 2023-03-19 15:59:08.194 [OpenSubtitles background worker 1] We have failed a lookup for /data/media/Video/Filme/Das Licht, aus dem die Träume sind.mkv so let's see if it was recent
I think it was because UMS had attempted to get the metadata for those videos before and failed for some reason.

In the latest I can see what UMS is doing when first attempting to get the metadata.

Code: Select all

DEBUG 2023-03-21 20:30:42.645 [OpenSubtitles background worker 1] OpenSubtitles hash for "Das Licht, aus dem die Träume sind.mkv" is c935ec150bcd7a1d
TRACE 2023-03-21 20:30:42.652 [OpenSubtitles background worker 1] Getting API data from: https://api.universalmediaserver.com/api/media/video/v2?osdbHash=c935ec150bcd7a1d&filebytesize=6406649958&language=de
DEBUG 2023-03-21 20:30:42.966 [OpenSubtitles background worker 1] API status was 404 for {"error":"Metadata not found."}
, https://api.universalmediaserver.com/api/media/video/v2?osdbHash=c935ec150bcd7a1d&filebytesize=6406649958&language=de
TRACE 2023-03-21 20:30:42.972 [OpenSubtitles background worker 1] Failed lookup for Das Licht, aus dem die Träume sind.mkv
And it is similar for most of your videos.

There is some that do get metadata you should be able to view now like the Addams Family that does look to have received metadata

Code: Select all

DEBUG 2023-03-21 20:30:56.288 [OpenSubtitles background worker 1] OpenSubtitles hash for "Addams Family S01E01.mp4" is 48f6495f209a1281
TRACE 2023-03-21 20:30:56.289 [OpenSubtitles background worker 1] Getting API data from: https://api.universalmediaserver.com/api/media/video/v2?title=Addams+Family&season=1&episode=1&osdbHash=48f6495f209a1281&filebytesize=202972930&language=de
DEBUG 2023-03-21 20:30:56.571 [OpenSubtitles background worker 1] API URL was https://api.universalmediaserver.com/api/media/video/v2?title=Addams+Family&season=1&episode=1&osdbHash=48f6495f209a1281&filebytesize=202972930&language=de
TRACE 2023-03-21 20:30:56.573 [OpenSubtitles background worker 1] Found an API match for Addams Family S01E01.mp4
TRACE 2023-03-21 20:30:56.574 [OpenSubtitles background worker 1] API data matches filename data for Addams Family S01E01.mp4
TRACE 2023-03-21 20:30:56.575 [OpenSubtitles background worker 1] Did not find title by IMDb ID using query: HikariProxyPreparedStatement@1631265463 wrapping prep5683: SELECT TV_SERIES.TITLE FROM TV_SERIES WHERE TV_SERIES.IMDBID = ? AND API_VERSION = ? LIMIT 1 {1: 'tt0057729', 2: '1-1-1'}
TRACE 2023-03-21 20:30:56.575 [OpenSubtitles background worker 1] API metadata for TV series Addams Family (IMDb ID: tt0057729) does not already exist in the database
TRACE 2023-03-21 20:30:56.576 [OpenSubtitles background worker 1] We have no failed lookups stored for addamsfamilytt0057729
TRACE 2023-03-21 20:30:56.578 [OpenSubtitles background worker 1] Getting API data from: https://api.universalmediaserver.com/api/media/series/v2?title=Addams+Family&imdbID=tt0057729&language=de
DEBUG 2023-03-21 20:30:56.917 [OpenSubtitles background worker 1] API URL was https://api.universalmediaserver.com/api/media/series/v2?title=Addams+Family&imdbID=tt0057729&language=de
DEBUG 2023-03-21 20:30:56.928 [OpenSubtitles background worker 1] not storing the series API lookup result because file and API TV series titles do not match. theaddamsfamily vs addamsfamily
TRACE 2023-03-21 20:30:56.928 [OpenSubtitles background worker 1] Searching for file/series in FAILED_LOOKUPS with "HikariProxyPreparedStatement@1776564433 wrapping prep5707: SELECT FAILED_LOOKUPS.FILENAME, FAILED_LOOKUPS.FAILUREDETAILS, FAILED_LOOKUPS.VERSION FROM FAILED_LOOKUPS WHERE FAILED_LOOKUPS.FILENAME = ? LIMIT 1 {1: 'addamsfamily'}" before update
TRACE 2023-03-21 20:30:57.364 [OpenSubtitles background worker 1] Converting byte array image source to JPEG format and type DLNAThumbnail using the following parameters: ScaleType = MAX, Width = 640, Height = 480, PadToSize = False
TRACE 2023-03-21 20:30:57.387 [OpenSubtitles background worker 1] Image conversion DLNA compliance check for JPEG_RES_300_225: The source image colors are compliant, format is non-compliant and resolution (300 x 225) is compliant.
Failures:
  JPEG DLNA compliance failed with too low JFIF version "1.01"
  JPEG DLNA compliance failed with illegal compression type "PROGRESSIVE_HUFFMAN"
  JPEG DLNA compliance failed with non-typical Huffman tables
TRACE 2023-03-21 20:30:57.397 [OpenSubtitles background worker 1] Finished converting 300x225 JPEG image. Output image resolution: 300x225, profile: JPEG_RES_300_225. Flags: Convert colors = False, Re-encode = True
TRACE 2023-03-21 20:30:57.397 [OpenSubtitles background worker 1] Setting episode name from api: The Addams Family Goes to School
TRACE 2023-03-21 20:30:57.402 [OpenSubtitles background worker 1] setting metadata for Addams Family S01E01.mp4
TRACE 2023-03-21 20:30:57.415 [OpenSubtitles background worker 1] Record already exists null 22 Carolyn Jones
TRACE 2023-03-21 20:30:57.416 [OpenSubtitles background worker 1] Record already exists null 22 John Astin
TRACE 2023-03-21 20:30:57.416 [OpenSubtitles background worker 1] Record already exists null 22 Allyn Joslyn
TRACE 2023-03-21 20:30:57.417 [OpenSubtitles background worker 1] Record already exists null 22 United States
TRACE 2023-03-21 20:30:57.418 [OpenSubtitles background worker 1] Record already exists null 22 Comedy
TRACE 2023-03-21 20:30:57.418 [OpenSubtitles background worker 1] Record already exists null 22 Family
TRACE 2023-03-21 20:30:57.418 [OpenSubtitles background worker 1] Record already exists null 22 Horror
TRACE 2023-03-21 20:30:57.419 [OpenSubtitles background worker 1] Record already exists null 22 https://m.media-amazon.com/images/M/MV5BNjQyOTYyMzY4Nl5BMl5BanBnXkFtZTgwMTIwNzc3MjE@._V1_SX300.jpg
TRACE 2023-03-21 20:30:57.420 [OpenSubtitles background worker 1] Record already exists null 22 7.8
TRACE 2023-03-21 20:30:57.421 [OpenSubtitles background worker 1] Record already exists null 22 net.pms.media.metadata.ApiRatingSource@7ec0004c
TRACE 2023-03-21 20:30:57.422 [OpenSubtitles background worker 1] Record already exists null 22 1964-09-18T00:00:00.000Z
TRACE 2023-03-21 20:30:57.423 [OpenSubtitles background worker 1] Found existing thumbnail with ID 35 in THUMBNAILS, setting the THUMBID in the FILES table
TRACE 2023-03-21 20:30:57.424 [OpenSubtitles background worker 1] THUMBID updated to 35 for /data/media/Video/Serien/Addams Family/Addams Family S01E01.mp4
But it may not store everything as I saw this too

Code: Select all

not storing the series API lookup result because file and API TV series titles do not match. theaddamsfamily vs addamsfamily
For some reason UMS can't get a match on the videos to get the right metadata, but looking at what it is searching for it should be finding matches. The only thing I can think of is it cannot find anything in German

Code: Select all

API status was 404 for {"error":"Metadata not found."}
, https://api.universalmediaserver.com/api/media/video/v2?osdbHash=c935ec150bcd7a1d&filebytesize=6406649958&language=de
This should be working for other languages so if not then it could be a bug. You could try setting the language to English and see if that makes a difference.

By going to that address in the web browser you can see exactly what UMS is getting. I tried changing the language to en-GB but it still could not match.
When trying with the Addams family I get something no matter the language so that may not be the cause.

I'm stumped at what else could cause the API to fail to match, maybe it is just limited to what it is searching. I think this is something for the Devs to investigate.

Have you changed those videos in same way like transcoding them to another format?
Logs are important for us to help, Please follow This Link before asking for support. Just a forum cleaner, Will help if I can but no expert.
psmoorf
Posts: 10
Joined: Sat Jan 21, 2023 11:15 pm

Re: Missing the metadata

Post by psmoorf »

I was testing a new installation with the actual version. With this I get metadata for the series in German and English. Only the movies get no one (most of them are uncompressed original blue-ray copies and downloads from tv mediathek).
If you want to have a look to this I create a new debug log.

With some additions to the config file it is possible to stream all content to my WebOS 2 TV too.
So I cross the finger to get better support for the metadata in the future.

Thanks for all the work you have with my problems.
Attachments
ums_debug_20230401.zip
(2.82 MiB) Downloaded 1323 times
User avatar
mik_s
Moderator
Posts: 1115
Joined: Wed Aug 23, 2017 11:03 pm
Location: UK

Re: Missing the metadata

Post by mik_s »

psmoorf wrote: Sun Apr 02, 2023 3:57 am Only the movies get no one (most of them are uncompressed original blue-ray copies and downloads from tv mediathek).
This might be the reason why the right metadata cannot be found.

I don't know exactly how this works but it appears that a hash is generated that can identify what that video is from its subtitles then this is used to get the metadata.
Another thing I saw that was required is the filesize. If these are your own rips or from sources not known by the API then it is unable to find an exact match.
I would assume just the hash would be enough to make a match but when I try it gives the error

Code: Select all

error	"filebytesize is required when passing osdbHash"
I guess the filesize is used for getting the right subs for various versions found online so they sync properly but would not have thought it to be required to retrieve other metadata.

I did post a message on GitHub for the devs to look into it and I confirmed those videos are found on https://www.themoviedb.org/ which is where the metadata is gathered from.
Logs are important for us to help, Please follow This Link before asking for support. Just a forum cleaner, Will help if I can but no expert.
User avatar
SubJunk
Lead Developer
Posts: 3705
Joined: Sun May 27, 2012 4:12 pm

Re: Missing the metadata

Post by SubJunk »

Thanks for these logs, and your great support and investigation mik_s, you are the best!

I think I can see a problem. UMS is sending an API request without the filename. I need to investigate and fix this.

In your logs it is sending: https://api.universalmediaserver.com/ap ... anguage=de which fails, but it should be sending
https://api.universalmediaserver.com/ap ... KkKlansman which works
User avatar
SubJunk
Lead Developer
Posts: 3705
Joined: Sun May 27, 2012 4:12 pm

Re: Missing the metadata

Post by SubJunk »

I have a fix candidate ready at https://github.com/UniversalMediaServer ... /pull/3882
I'm not able to upload it for testing yet but will within the next few days, unless someone else does that first
User avatar
SubJunk
Lead Developer
Posts: 3705
Joined: Sun May 27, 2012 4:12 pm

Re: Missing the metadata

Post by SubJunk »

There's a fix available to test at https://www.universalmediaserver.com/up ... tadata.exe
I have tested it with the same filename as your logs and it worked, will be great if you can confirm that.
To test this, you will need to delete or rename your "C:\ProgramData\UMS\database" directory, because that database makes sure that failed attempts are not retried for 7 days (and here we need them to retry)
If you have important info in your database, like which videos have been watched, you can restore the old "database" folder after testing. Then I will upload a build that will automatically fetch the failed data
psmoorf
Posts: 10
Joined: Sat Jan 21, 2023 11:15 pm

Re: Missing the metadata

Post by psmoorf »

I now tested the test version on a virtual machine with windows 10.
The result you can see inside the log. With this configuration in the attachment I did not get any metadata.

Thanks again for your work.
Attachments
ums_dbg_2023-04-22-17-34.zip
(614.7 KiB) Downloaded 1301 times
User avatar
mik_s
Moderator
Posts: 1115
Joined: Wed Aug 23, 2017 11:03 pm
Location: UK

Re: Missing the metadata

Post by mik_s »

From what I can see in the log the metadata was retrieved and up to date, however I think those files were scanned before you put UMS into trace mode so I can not see the process.

Code: Select all

net.pms.util.APIUtils The latest metadata already exists for BlacKkKlansman.mkv
net.pms.util.APIUtils The latest metadata already exists for Das Licht, aus dem die Träume sind.mkv
net.pms.util.APIUtils The latest metadata already exists for Der beste Film aller Zeiten.mkv
net.pms.util.APIUtils The latest metadata already exists for Der See der wilden Gänse.mp4
net.pms.util.APIUtils The latest metadata already exists for Die Brücke von Arnheim.mp4
Could you try again but make sure UMS in in trace mode first then quit and delete the database.
Might be better putting UMS in trace mode by adding this to UMS.conf before you run it for the first time.

Code: Select all

log_level =TRACE


What does it look like in the web browser for this version? Could you post screenshots again?

I'm not sure if running in a VM might interfere with this too so could you try as your main installation?
If you want to keep all your settings and database then make a backup of C:\ProgramData\UMS then do a fresh install of that test version.
Logs are important for us to help, Please follow This Link before asking for support. Just a forum cleaner, Will help if I can but no expert.
User avatar
SubJunk
Lead Developer
Posts: 3705
Joined: Sun May 27, 2012 4:12 pm

Re: Missing the metadata

Post by SubJunk »

I'm marking this as solved, feel free to let us know if it isn't
Post Reply