mencoder half-green screen issue with x265 encodings and overscan compensation

[malcalevak]

I've been experiencing a reoccurring problem whenever I try to play x265 encoded media via UMS, when mencoder is used w/ overscan compensation (8% on height and width), where the resulting output is literally half of a green screen, as seen here:

half-green.jpg (72.21 KiB) Viewed 1430 times

I'm connecting to UMS via a PS4 Pro, though the issue still occurs if I use the same parameters to output to a file and play that on my PC. I have not encountered this issue with x264 encoded media, it's always been x265 encoded media.

Unfortunately, my TV has some pretty crazy overscan that can't be adjusted, hence the use of the compensation. This was actually the second half of a larger problem that miraculously seemed to resolve itself when I restarted UMS to create Trace logs. The first issue was that ffmpeg was failing to transcode the video with subtitles (it had no issues without), so it was then falling to mencoder, which had the above described issue. Since it's no longer creating the first issue, I won't worry about that, for now.

While investigating the issue, it does look like it's tied to the overscan compensation, when I removed the `expand` property from `-vf`the output file is fine.

Logs are attached.

[mik_s]

I don't know much about the actual transcoding command so can't tell if anything is wrong with that.

Ideally you want to avoid overscan compensation by having your TV to the right picture size where it is not needed.
On my Samsung I have it on "Screen Fit". There must be a similar option on yours. What make and model is your TV?

There are options in the PS4 settings that should fix this if you can't do it on your TV, then you won't need to add compensation in UMS.
Look in "Sound and screen" for "Display Area Settings" and you can move the box to what can be displayed.

One thing I noticed that may be causing the problems is the network speed to your PS4 is only 1.7mb/s.

Code: Select all

Renderer PlayStation 4 has an estimated network speed of 1.7160480551800499 Mb/s

This is limiting the transcoding that is possible and may be the cause of your other problems too. All your videos are marked to transcode even though they are supported to stream because the bitrate is too high for your network.
Some renderers need the bitrate to be half of the network's maximum bitrate in order to have smooth playback, and this option is set in the conf for the PS4 which only gives you 0.85mb/s for your PS4.

Code: Select all

Using video bitrate limit from PlayStation 4 configuration (1 Mb/s) because it is lower than the general configuration bitrate limit (90 Mb/s)
File "La.mpg" will not be streamed because the bitrate (5791487 b/s) is too high (0 b/s).

How is your network set up? Is the PS4/computer connected to the network wirelessly?

[malcalevak]

When it comes to the transcoding command, the one in question is:

DEBUG 2023-07-21 20:46:37.066 [mencoder.exe-243] net.pms.io.ProcessWrapperImpl Starting "C:\Program Files (x86)\Universal Media Server\bin\mencoder.exe" -ss 0 D:\Video\Movies\test.mkv -msglevel statusline=2 -oac lavc -of lavf -lavfopts format=mpegts -ovc x264 -channels 2 -lavdopts debug=0:threads=4 -lavcopts autoaspect=1:acodec=ac3:abitrate=448:threads=8 -x264encopts crf=23:preset=superfast:level=31:threads=auto:vbv_maxrate=0:vbv_bufsize=1000 -ass -ass-color FFFFFF00 -ass-border-color 00000000 -ass-font-scale 1.4 -font C:\Windows\Fonts\Arial.ttf -ass-force-style FontName=Arial,Outline=1,Shadow=1,MarginV=50 -fontconfig -noautosub -sid 1 -ofps 24000/1001 -vf softskip,expand=-152:-80,scale=1920:1016,harddup -lavdopts fast -mc 0 -noskip -af lavcresample=48000 -srate 48000 -o \\.\pipe\mencoder1689986797050

Removing the `expand=-152:-80` seems to resolve the issue, but removes the overscan compensation. I've come to the conclusion this is actually a mencoder issue, because the exact same command works on an x264 file, but causes the half-green screen issue with x265. I'm not sure where to follow up with them about issues, though.

Getting to what you were saying about overscan compensation, I'd love to avoid it, but I've spent hours looking in the past, and this TV is just bad, lol. It's a Samsung HL-T5075S, it's actually pretty old, uses DLP tech, technically only does 1080i/720p. I've looked high and low, and all I can find is that it's got really bad overscan, and there doesn't seem to be any fix, short of adjusting the projector's physical location, which would cost more money that it's worth. FWIW, here's a link to the manual: https://mans.io/files/viewer/14697/29

As for the PS4 settings you mentioned, that don't seem to do anything useful. I'd previously adjusted it as small as it will go, and I can see the lines in all the corners, but it doesn't appear to apply to UMS. It doesn't apply to Hulu either, which for some reason positions subtitles differently than other apps, causing them to frequently run off the bottom of the screen. I'm attaching 3 screenshots, the first is the PS4 setting you'd mentioned, you'll see it's as small as it can get. Next, I have two runs of the same test pattern, once without overscan compensation, and once with. You can see the difference - both are with the PS4 setting as small as it will go.

Speaking of which, I have an override for FFMPEG set in a configuration file specific to my PS4, but it currently doesn't seem to be working... I haven't yet had a chance to mess around with it.

I would love to be proven wrong about either of those two things! I'm mostly just waiting for the TV to go bad so I'm forced to replace it.

I thought maybe you were on to something with the network speed, unfortunately, after finding a cable long enough, it doesn't seem to be making any sort of difference. I'm also unsure of how reliable the UMS estimated speed is. I tested the PS4 network connectivity after switching to wired, and it was pulling 220Mbps down, but UMS is seeing only 50Mbps, on a good estimate. Regardless, trying the same x265 encoded file will frequently still fail and fall to mencoder, leading to the half-green screen. Usually, I can get around it, by manually specificing I want to use FFMPEG, and retrying after it fails the first time. I'm wondering if for some reason FFMPEG is taking a split second too long during transcoding, leading it to think it's failed?

PS4 Overscan setting:

PXL_20230727_192933375[1].jpg (2.53 MiB) Viewed 1364 times

W/o Overscan Compensation:

PXL_20230727_192458853.MP[1].jpg (4.67 MiB) Viewed 1364 times

With Overscan Compensation:

PXL_20230727_192434227[1].jpg (2.29 MiB) Viewed 1364 times

[malcalevak]

Brief addendum, the mencoder issue seems to be specifically tied to 10-bit x265 encodings. I ran the test pattern through ffmpeg and encoded 3 files, all HEVC, one 8-bit, one 10-bit, one 12-bit.

• The 8-bit file played perfectly

• The 10-bit file did the half-green screen, I also noticed the horizontal overscan compensation wasn't correct on the half it did render.

• The 12-bit file errored out, which I'm not currently concerned with, but thought I'd note

If you happen to know where to file issues for mencoder, I'll gladly follow up with them about it.

[mik_s]

I think the screen mode you want is "Wide Fit" but being a DLP TV I don't think you can avoid the overscan unless, like you say, you can reposition the projector.
It may actually be possible to do in the TVs service mode but could mess up alignment. I did find the service manual for it here. Using an adblocker I was able to get rid of the popup saying you need to buy it to view.

I did a little searching and found this

I have since found that in order to adjust for overscan by renaming HDMI1 to PC the input source must be 720p, not 1080i.

That was just one page in a long thread so there may be better solutions if you have a look through.

I think I have seen somewhere that mencoder has a problem with 10 bit, but it may have been HDR related. I don't know if 12 bit is supported.

It may be better to use FFmpeg instead and can add custom options to the transcoding command from the renderers conf file to add padding that would add the compensate for the overscan. This is from the defaultrender.conf

Code: Select all

# Overrides the FFmpeg custom options in UMS for this renderer. Default is
# empty, which means the UMS settings will be used.
# All setting for output formats, codecs, bitrate e.g. can be used except video,
# audio and subs mapping (-map).
# Do not use it to override -vf; use OverrideFFmpegVideoFilter instead.
# Default: ""
CustomFFmpegOptions =

I don't know what options to use or how, its something you will have to experiment with. I did find this that may be helpful or at least give you an idea on what to look for. There is a test video there you can use to find out exactly how much overscan you have.
This might be better for a general formula to add padding.

You will have to make sure your modded conf is being used though. I usually add -mod to the renderer name and the .conf filename so I know if it is being used and it will not be overwritten when you update UMS. I also set "LoadingPriority =" to a higher number so that it will take precedent over the original conf.

One downside of adding this padding is that every video, even supported ones, will have to be transcoded and not streamed. On some renderers this could mean loosing the ability to seek but the PS4 can handle this. Shame the PS4 setting for overscan does not work in apps, it may be only for games.

If you are using a custom conf made from the option in the old GUI I think it may be broken in recent versions, it might be fixed now but may not be in the current public release yet. Best to keep your conf in the main renderers folder where UMS is installed.

For your network speed it might have been due to eSET blocking things so UMS was unable to get the correct speeds.
Now you have that sorted out if you still have problems make new logs and I might be able to see something.

[malcalevak]

Thanks for the research! I vaguely recall seeing something about renaming HDMI to PC, but never hard luck with it. That said, I'll dig around and see if I have better luck this time. I'd definitely love to be able to disable it!

Using FFmpeg is my preference, the only reason mencoder was entering the equation was because for some reason FFmpeg kept failing initially (but multiple attempts, or manually specifying it in the transcode folder seemed to resolve it). I guess I can just disable it entirely.

I've actually been using the following command to compensate for overscan:

Code: Select all

OverrideFFmpegVideoFilter = scale='if(gt(iw,1760)+gt(ih,990), (iw*0.91)*sar, iw*sar)':'if(gt(iw,1760)+gt(ih,990), (ih*0.91), ih)', pad=max(iw*1.09\,ih*1.09*(16/9)):ow/(16/9):(ow-iw)/2:(oh-ih)/2

It seemed to do the job, it's only recently that I've noticed it doesn't appear to be working. The log is showing

Code: Select all

Loaded device configuration PlayStation4-10.0.0.4.conf for 10.0.0.4

Which at least implies it's loading, but I'm not sure of how to confirm it. I know using the old GUI, trying to open the specific configuration actually just tries to make a new file, and I don't see a way to open/edit these files via the web based GUI. I could try putting it in the main folder like you said.

eSET doesn't appear to have anything to do with the network speeds. I'll try to make some new logs tomorrow. Thanks again!

[mik_s]

If you were using an IP address instead of a UUID in that custom conf then due to a bug in recent versions UMS will ignore it.
Someone else was using IPs in custom confs to identify which renderer was which as they did not use a UUID, but it stopped working for them too after an update. You can read about it here

It has just be fixed and confirmed working but you will have to wait a while until a proper release as I don't think that test version is still online, or if it is probably won't be for long as it has been marked solved.

I personally prefer to make a new conf and just put it in the main renderers folder so I know it will be used.

[malcalevak]

Good news! The renaming of HDM1 did the trick, the TV no longer has massive overscan! Unfortunately, from what I was reading, this does mean it's doing some scaling, but it might be the best I can get until I replace it, so thank you for the assist!

Because I'm no longer scaling anything, I shouldn't actually encounter the issue with 10-bit x265 encodings and mencoder, nor does it matter that I can't get the custom conf to load.

I think I might still reach out to the mencoder team, to at least make them aware of the issue, but it's clearly not a UMS issue.

Even though I'm not using it anymore, it might still warrant a look to see if settings are actually being applied when custom confs are loaded (it said the file was loaded, but the result implied it wasn't).

Anyway, this leaves me with the network speed issues. As you can see here, I do a speed test on the PS4 and get 171.2Mbps, to the internet, which should be slower than what I get within the now wired-network, but UMS is estimating 13.61310204366683 Mb/s. I've attached new debug logs, to see if you can lend any solutions there. For additional information, the PC UMS is running on is a hardwired desktop. A speedtest I just ran, which is going through a VPN, is 224.41Mbps. I'm not sure how to do a speedtest within the network, but it should be substantially greater.

I don't imagine that this is related, but I've also noticed the logs showing:

Code: Select all

13:18:57 INFO  VSFilter / DirectVobSub was not found. This can cause problems when trying to play subtitled videos with AviSynth.

I went ahead and manually installed the VSFilter compatible XySubFilter, but it doesn't appear to be making a difference.

[mik_s]

Glad that worked, its much less hassle if you can deal with overscan on the TV itself. If is is doing any scaling then the change in quality might be just about the same as transcoding so hopefully you should not notice any difference. When I had my old Plasma I could not tell the difference between 720p and 1080i.

Your network speed to the PS4 has improved but I don't know why it is so low. Something must still be blocking some traffic.
While I was looking at the speed results I noticed that an unknown renderer on 10.0.0.10 is showing as NaN as there is no reply to the ping.
From what I can tell that IP is that of your computer, usually windows media player, and should have a very high speed (I get over 700Mb/s) as it is basically a loopback test.
There is another unknown renderer on 10.0.0.7 which looks to be Serviio which does have a good speed test with over 90Mb/s. I don't know anything about that except it is a similar media server so it maybe interfering with UMS, though being on a different computer it shouldn't.

I still suspect eSET is doing something but no idea what to look for.
I've tried to search for any other ports that may be used and found references to ports 1044, 5002, and UDP 1900 in various places but no concrete information on them or if they are being used. All I know for certain is that by default UMS uses 9001 for the web GUI, 9002 for the standalone web player, and 5001 for the server. These can be changed in the settings though.

For your custom conf not working I am sure it is because of that bug where it not using the renderers IP. Try to see if the test version is still active in that link in my last post.
As that conf was not being used it was not added to the zip file so can't tell if anything is wrong with it, if you attach it separately I'll give it a look. I can see it being loaded in your previous logs but not in this one.

I would not worry about the filters being missing as that will only apply to AviSynth which you are not using. This is something I know nothing about though.

[malcalevak]

Thanks, I'll experiment some more to see if I can hone in on what might be causing the speed issues.

When it comes to the custom conf, I completely missed your earlier reply! That's exactly the issue, it was using the IP address (this is how it creates them by default). I remember trying to do it by UUID before, but having no luck. More so out of curiosity, how do you find a devices UUID?