Wikipedia:Reference desk/Archives/Computing/2013 March 6

Welcome to the Wikipedia Computing Reference Desk Archives
The page you are currently viewing is an archive page. While you can leave answers for any questions shown below, please ask new questions on one of the current reference desk pages.

March 6

Video codecs for future storage

Part 1
I'm getting several hours of super 8 film digitized. I want to keep it in a format that strikes a good balance between keeping the amount of storage space needed at a managable level, and preserving as much as possible of the information in the video. I've had tests done at several companies who specialize in this, and the company so far that produces the best digitizations, uses this hardware, and a codec that VideoLan describes thusly:

Stream: 0
Type: Video
Codec: Packed YUV 4:2:2, U:Y:V:V (2vuy)
Language: English
Resolution: 1280x720
Display resolution: 1280x720
Frame rate: 24

edit: YUV 4:4:2 corrected to YUV 4:2:2

When I calculate the total number of pixels encoded, the file size is about 2.7 bytes per pixel.

• Do we have an article about this codec? I've read Chroma subsampling, but anything more specific?
• Is this a codec that can be expected to be readable a long time into the future?
• In this particular codec, are all frames represented equally faithful to the original, or are there key frames that are represented more faithfully, with diffs in between which use some kind of lossy compression with respect to the key frames? To clarify: The details in the encoding scheme, i.e. whether key frames and diffs are used or whether information is stored a sequence of entire images, does not matter to me. If the diffs are exact, it's fine.
• I'm happy with a file size of up to about 4 bytes per pixel. Are there codecs that are widely used, can be expected to be readable a long time in the future, and which can be manipulated with ffmpeg (Windows version from http://ffmpeg.zeranoe.com/builds/), that would be a better choice than the one described above?

Part 2

• For some reason, the company mentioned above only delivers the files in .MOV containers. Can I convert to .AVI losslessly with ffmpeg, and safely delete the original?
• Will ffmpeg be able to split longer files into shorter sequences losslessly?

Thanks, --NorwegianBlue ^talk 07:42, 6 March 2013 (UTC)

1a. I can't find a specific article but YUV has more information. 1b. Yes. 1c. It is uncompressed; you lose nothing except half the horizontal color resolution, which is normally not noticeable and which virtually all other codecs lose as well. Every frame is a key frame. 1d. 8-bit YUV 4:2:2 is two bytes per pixel. You could ask about 24-bit RGB, which would be 3 bytes per pixel, but it's possible that YUV 4:2:2 is the raw output format of their digitizer and converting to anything else would lose information. You could also ask about more than 8 bits per channel, but I think you would have more compatibility problems. 2a. I think ffmpeg -i input.mov -vcodec copy output.avi will work. 2b. Probably. At any rate there are tools that can. -- BenRG (talk) 18:47, 6 March 2013 (UTC)

Thanks a lot, BenRG. I wonder what the extra 0.7 bytes per pixel are? I just discovered Mediainfo, which reports the format like this:

Extended content

General Complete name  : MyFile.mov Format  : MPEG-4 Format profile  : QuickTime Codec ID  : qt File size  : 8.41 GiB Duration  : 3mn 23s Overall bit rate  : 355 Mbps Encoded date  : UTC 2013-01-24 16:33:18 Tagged date  : UTC 2013-01-24 16:36:06 Writing library  : Apple QuickTime ©TIM  : 00:00:00:00 ©TSC  : 24 ©TSZ  : 1 Video ID  : 1 Format  : YUV Codec ID  : 2vuy Duration  : 3mn 23s Bit rate mode  : Constant Bit rate  : 354 Mbps Width  : 1 280 pixels Height  : 720 pixels Display aspect ratio  : 16:9 Frame rate mode  : Constant Frame rate  : 24.000 fps Color space  : YUV Chroma subsampling  : 4:2:2 Compression mode  : Lossless Bits/(Pixel*Frame)  : 16.000 Stream size  : 8.37 GiB (100%) Title  : Apple aliasdatahåndterer Language  : English Encoded date  : UTC 2013-01-24 16:33:18 Tagged date  : UTC 2013-01-24 16:36:06 Audio ID  : 2 Format  : PCM Format settings, Endianness  : Little Format settings, Sign  : Signed Codec ID  : sowt Duration  : 3mn 23s Bit rate mode  : Constant Bit rate  : 1 536 Kbps Channel(s)  : 2 channels Channel positions  : Front: L R Sampling rate  : 48.0 KHz Bit depth  : 16 bits Stream size  : 37.2 MiB (0%) Title  : Apple lydmediahåndterer / Apple aliasdatahåndterer Language  : English Encoded date  : UTC 2013-01-24 16:33:18 Tagged date  : UTC 2013-01-24 16:36:06 Other ID  : 3 Type  : Time code Format  : QuickTime TC Duration  : 3mn 23s Time code of first frame  : 00:00:00:00 Time code settings  : Striped Title  : Tidskodemediehåndterer / Apple aliasdatahåndterer Language  : English Encoded date  : UTC 2013-01-24 16:36:06 Tagged date  : UTC 2013-01-24 16:36:06

Could a silent audio stream consume so much space? I interpret the "Bits/(Pixel*Frame) : 16.000" output as a confirmation of your statement that the yuv compression should result in 2 bytes per pixel.

The conversion to .AVI that you suggested worked, sort of. The file played fine in VideoLan, I compared a PNG snapshot with the corresponding frame in the original, and found the the files to be identical (binary diff). However, this particular content in an AVI container crashed MediaInfo (but not ffprobe). --NorwegianBlue ^talk 23:27, 6 March 2013 (UTC)

You don't need a silent audio track whether it consumes a lot of space or not, include -an in your ffmpeg command. ¦ Reisio (talk) 00:09, 7 March 2013 (UTC)

I know I don't need it, but I'm puzzled by the size of the files as delivered by the company who did the digitization. I don't want to mess with the original files until I'm 110% sure what I'm doing, but I'll try stripping it as you suggested on a copy of one of the files, to see if that's the explanation. Thanks! --NorwegianBlue ^talk 08:41, 7 March 2013 (UTC)

1280×720×24×16 = 353894400, which is consistent with the 354 Mbps rate quoted above, and 203s×354Mbps = 8.37 GB, so I think either you miscalculated or the file with the problem is a different one. Note that 1 Mbps = 10⁶ bps, but 1 GB = 2³⁰ B. The audio track accounts for less than 1% of the file size, as shown by Mediainfo. It's possible they include it because some playback software wasn't designed to handle files without an audio track. You could at least downsample it to 8 KHz or something. -- BenRG (talk) 18:31, 7 March 2013 (UTC)

I indeed miscalculated, and was about to come here and say, slightly embarrassed, "mystery solved". I used the frame rate of the source material (18 fps), which was part of the file names, and completely ignored the blindingly obvious fact that they had inserted extra frames, to adjust to 24 fps instead of 18 fps. But it's a good thing you beat me to it, since in doing so, you also spelled out how other relevant calculations, which I was unsure about, are done. Thanks a lot! --NorwegianBlue ^talk 20:23, 7 March 2013 (UTC)

How to simultaneously copy the text of all the alternatives in a Combo box (drop-down menu) on a web page?

Unresolved

I use Windows7 and Firefox v19 and/or Microsoft Internet Explorer v9.
I need to copy the text of all the available options inn a Combo box ("drop-down list") and paste them into my local text editor.
For example I would like to simultaneously copy all the options:

Volvo
Saab
Opel
Audi

from the combo box in the Result field on the web page: http://www.w3schools.com/tags/tryit.asp?filename=tryhtml_select (Link found and visited: 2013-03-06)
(Of course: In most cases there are tens or sometimes maybe hundreds of options in the combo boxes that I need to copy).
How may I do this?
--46.15.136.227 (talk) 17:20, 6 March 2013 (UTC)

(OP) Adding to my question at 23:21, 6 March 2013 (UTC) :
The example list (Volvo Saab Opel Audi), referred to above, only uses the HTML-tag: <select>…, but of course I also sometimes need to copy a combo box-list that is given in a script, like the one on the web page http://www.oft.gov.uk/ (visited: 2013-03-06) under: "I am looking for..." (with the options: "Anti-money laundering", "OFT Board", "Current Campaigns", …).
The solutions suggested, below, so far (Thank you all!), namely:
1. Screenshot and OCR.
2. Viewing the page source, searching, cuting and pasting (with or without the Firefox Firebug-Add-on).
they both do work, but have prooven to be far too time consuming and cumbersome in most cases.
I guess, what I wish for is the ability to just riht-click the combo box-arrow and choose: "Copy the whole selection list to the clipboard".
-- (OP) 46.15.136.227 (talk) 23:21, 6 March 2013 (UTC)

If all you have access to is the finished app, try this: It's messy, but a screen grab (or multiples, if it doesn't fit on one page) followed to OCR could do it. If you have access to the source code, then better methods are available. StuRat (talk) 18:06, 6 March 2013 (UTC)

You could just view the page source and copy the options and their surrounding HTML, which you would then of course have to edit out, either by some suitable searching-and-replacing in a text editor, or perhaps a spreadsheet if there are extra complications. AndrewWTaylor (talk) 19:41, 6 March 2013 (UTC)

I do stuff like that pretty often. Usually (? always? depends what browser I guess) there's a "view source" via a menu or rightclick or something. When you've got the HTML source, if you're lucky, you can search for one of the combo box entries and find the place where they're all specified. If you're not lucky, they might be elsewhere in some CSS file or something that's transcluded into the HTML, in which case I'm out of my depth and somebody else might be able to continue the advice. Gzuckier (talk) 19:46, 6 March 2013 (UTC)

If you use Firebug (a Firefox addon) it's a simple matter of rightclicking the select box, "Inspect with FireBug", copy-pasting the source code there, and getting rid off the tags using search-and-replace. The inspect will work whether or not the dropdown was in the original source code, or generated by JavaScript. 82.75.247.59 (talk) 20:01, 6 March 2013 (UTC)

I have expanded my original question a little (see above, just below my first entry).
-- (OP) 46.15.136.227 (talk) 23:21, 6 March 2013 (UTC)

On a page with a lot of OPTION tags like that, you can paste something like this into the address bar of your browser:

javascript:var o=document.getElementsByTagName("option"); var r=""; for(var i=0;i<o.length;i++) { r+=o[i].innerHTML+"\n"; }; alert(r);

In theory, this will then give you a pop-up box with all of the text between the OPTION tags. You should then (on a modern browser, like Chrome) be able to copy those directly and paste them into your text editor. Note that this won't work exactly on that w3schools page, because those OPTION tags are embedded in an IFRAME. --Mr.98 (talk) 23:55, 6 March 2013 (UTC)

If you did need to go inside of an IFRAME like the one on that page — or if you just want to experiment with it — you could use this code instead:

javascript:var o=document.getElementsByTagName("iframe")[1].contentWindow.document.getElementsByTagName("option"); var r=""; for(var i=0;i<o.length;i++) { r+=o[i].innerHTML+"\n"; }; alert(r);

You would have to modify the number after the "iframe" (1 in this case) to refer to the right one (there are at least two IFRAMEs on that page, so in this case it is accessing the second one, because these are ordered starting from 0). --Mr.98 (talk) 00:03, 7 March 2013 (UTC)

openadserving.com malware

had this infesting me for a while now, but norton (as supplied by my isp and automatically updated) doesn't seem to do anything with it. I got it temporarily bypassed with the hosts file but would sure like to eradicate it from the system, but a quick googling comes up with a whole mess of different lists of files, registry keys, startup menu entries to delete, i.e. a different list from every source. So, does anyone know when any of the antivirus guys are going to get a handle on this, or why they can't? Gzuckier (talk) 21:28, 6 March 2013 (UTC)

Have you tried Malwarebytes in safe mode? It's saved me from reinstalling Windows many times. I don't know if it will work in your case or not, but it's worth a shot. And from my experience/reading, Norton really isn't all that great - I would recommend Microsoft Security Essentials for real-time protection. Both AV's I've linked are free. ProtossPylon 00:00, 7 March 2013 (UTC)

This is just my personal opinion, but I'm not sure I'd call MalwareBytes an antivirus program.--Rockfang (talk) 18:33, 7 March 2013 (UTC)

What would you call it, then? That's the only description I can think of for it. ProtossPylon 23:10, 7 March 2013 (UTC)

Unix question

Hi. I have to survey a network of computers; there are Windows boxes, Mac boxes, various flavours of linux, Ubuntu and Kubuntu. One thing I need to do is establish how many hard disks (not partitons, actual physical drives) are inside each box. This is easy peasy for the Windows and Mac ones, but is there a simple fast and dirty way of doing it via a terminal on the others? I have some (but admittedly not a lot) experience of using bash. Cheers  pablo 23:40, 6 March 2013 (UTC)

fdisk -l | grep 'Disk /' (as root / via sudo), /sbin/blkid, million other ways. :) ¦ Reisio (talk) 23:53, 6 March 2013 (UTC)

And the answer is fdisk: illegal option -- l ... maybe I have less experience than I thought, or need a more foolproof solution! pablo 00:31, 7 March 2013 (UTC)

That's odd. /sbin/fdisk is part of util-linux, so it should work on any desktop or server Linux. Embedded Linux distributions may use the Busybox version instead, but even has the -l option too. -- Finlay McWalterჷTalk 00:42, 7 March 2013 (UTC)

The "fdisk -l" one is questionable, as it identifies 24 out of my fileserver's 11 hard disks: the 11 physical drives, the three software RAID arrays built from those drives, and the 10 LVM logical volumes sliced out of the RAID arrays. --Carnildo (talk) 01:50, 7 March 2013 (UTC)

You need admin access to run "fdisk -l" in most cases. You can try "sudo fdisk -l" if you're setup for it. Shadowjams (talk) 02:18, 7 March 2013 (UTC)

I seem to recall some oddity about Unix that you had to mount a disk to see it. Is that still the case ? StuRat (talk) 02:33, 7 March 2013 (UTC)

Not to see that it exists, or to see its partitions. Shadowjams (talk) 03:02, 7 March 2013 (UTC)

No. In order to mount an HDD or partition, the OS has to know it exists. You can type cat /proc/partitions and see both mounted and unmounted hard drives and partitions. Hard disks in Linux are listed in /dev. For example, if I had two SATA HDDs in my computer, I would have /dev/sda and /dev/sdb. Each disk is divided into one or more partitions. So if sda were divided into two partitions, I would have /dev/sda1 and /dev/sda2. If they were IDE hard drives, they would be /dev/hda and /dev/hdb. However, as Carnildo noted, a RAID controller can be used to group multiple disks into one and lead Linux to believe that the array is a single disk. For example, the RAID may have three disks, each 500 GB in size. But they're in a RAID 0, so Linux thinks they're one 1.5 TB disk. You may think that's an academic exception, but it's actually common in server environments where Linux is often used. In such a case, you should use the command-line utility that came with your RAID card to determine how many disks are in the array. For example, if your RAID is made by 3ware, you use tw_cli. If it's made by LSI, you use megamgr. If you're not using a RAID (which most home users aren't) then you can just ignore that part about RAIDs and simply count the number of lettered disks (e.g., sda, sdb, etc.) to get the number of disks in your system. Reading the contents of /proc/partitions is my preferred way to do so.—Best Dog Ever (talk) 03:21, 7 March 2013 (UTC)

Thanks all. Reading /proc/partitions has sorted things out in most cases, but on a few machines /proc/partitions does not exist, don't know why. fdisk does not run with the -l parameter on these machines either. /dev contains many, many entries of the form /sd??, for example

sd10q sd10r sd10s sd10t sd129i sd129j sd129k sd129l sd129m sd129n sd129o sd129p sd130q sd130r sd130s sd130t sd130u sd136a sd136b sd136c sd136d sd136e sd136f sd136g sd136h sd137i sd137j sd137k sd137l sd137m sd137n sd137o sd137p sd138q sd138r sd138s sd138t sd138u sd144a sd144b sd144c sd144d sd144e sd144f sd144g sd144h sd145i sd145j sd145k sd145l sd145m sd145n sd145o sd145p sd146q sd146r sd146s sd146t sd146u sd16a sd16b sd16c sd16d sd16e sd16f sd16g sd16h sd17i sd17j sd17k sd17l sd17m sd17n sd17o sd17p sd18q sd18r sd18s sd18t sd18u sd1i sd1j sd1k sd1l sd1m sd1n sd1o sd1p sd2q sd2r sd2s sd2t sd2u sd3a sd3b sd3c sd3d sd3e sd3f sd3g sd3h sd8a sd8b sd8c sd8d sd8e sd8f sd8g sd8h sd9i sd9j sd9k sd9l sd9m sd9n sd9o sd9p

from this it seems as though these partitions all exist on 18 physical hard disks. Am I interpreting this correctly? It does't seem likely as the box doesn't look big enough! pablo 16:27, 7 March 2013 (UTC)

If you type df does it list any physical devices in /dev under filesystem? What kind of box is it and what version of UNIX or Linux is it running? You can check by typing cat /proc/version or uname -a. I know BSD uses /dev/ada0, /dev/ada1, etc. Macs use /dev/disk0, /dev/disk1, etc.--Best Dog Ever (talk) 01:12, 8 March 2013 (UTC)

Under Linux, "df" lists filesystems, not drives, and only those filesystems that are mounted. Further confounding the issue is that it lists loopback filesystems, which have a "/dev" entry but which correspond to a file. --Carnildo (talk) 02:38, 8 March 2013 (UTC)

He can still use it to deduce which devices are being mounted as disks. In the left-column, it will list physical partitions currently mounted and he can then deduce what types of devices are being partitioned.—Best Dog Ever (talk) 03:24, 8 March 2013 (UTC)

Aha! Mystery boxes are running SunOS. Entries in dev/dsk are of the format c0t0d0s0. I have d0, d1 and d2 = 3 disks which seems more feasible. Apologies if I made this more complicated than it is, I now have a solution for each of the different operating systems. Thanks! pablo 09:27, 8 March 2013 (UTC)