Talk:Record-oriented filesystem

Is this article about filesystems or about file formats?

I'm confused about this article, is it a FILESYSTEM (like FAT, ext2, ReiserFS) or a FILE-FORMAT (magic numbers, file extension) system? Improfane

A: It's not about a specific file system, but rather the whole class of filesystems that support record-oriented operation. The key point is that the system calls used to access files are designed to access records, rather than chunks of data read or written in application-specific formats. Most mainframe operating systems support a rich variety of record-oriented record formats. Most commonly, records are fixed in length within any given file, or a file may have variable-length records. Unlike the stream-oriented systems found on systems like Unix, PC-DOS, Windows, and Mac, the data in the file is accessed strictly in terms of records. Variable-length records are preceded by a (usually) binary byte-count, and may contain any coded bytes at all, both binary and characters. There is no concept of an "end of line" delimiter, such as a carriage-return character.

Some people, particularly Unix advocates, dismiss record-oriented file systems as being based on punched-card technology, and therefore presumably "old-fashioned." The Unix-like stream-oriented approach is modelled after another 19th century technology, that of the paper-tapes used by the printing telegraph, used to mechanize the transmission of telegrams. These started being used for computers in the form of Teletype machines used as inexpensive input devices by the mini-computers of the '60s and '70s.

For its part, the Hollerith punched card was at least originally conceived for computational purposes.

This article, it seems to me, was written by a Unix advocate who wished to diminish the advantages of record-oriented file access methods. It is clearly not NPOV. I plan to fix it, when I find time to address the matter properly.

--RussHolsclaw 04:23, 12 February 2006 (UTC)

I agree! Moreover, terminology: does an IBM mainframe OS even use a filesystem? You have VTOCs, catalogs, data sets, but file system? Never heard of that. Source needed. --Kubanczyk 09:05, 7 October 2007 (UTC)

IBM calls file systems Access Methods.

Paper tape as used for text message transmission actually contain individual records which are delimited by various control characters. Each line of text (aka record) is terminated be a carriage-return character (which sends the print head to the left) and a line-feed character which rolls the paper platen up a line in position for the next line.

A better example of a datastream used in punched tape is in a numerical controlled machine tools NC These use a stream of commands to define which cutting tool to use, the starting position, subsequent points along the cutting path and other control information.

A record oriented file has several advantages. After a program writes a collection of data as a record the program that reads that record has the understanding of that data as a collection. Although it is permitted to read only the beginning of a record, the next sequential read returns the next collection of data (record) that the writer intended to be grouped together. Another advantage is that the record has a length and there is no restriction on the bit patterns composing the data record, i.e. there is no delimiter character.

There is a cost associated with record oriented. The length definition takes up space. On a magnetic tape that definition takes the form of an inter-record gap. On a disk a meta data area must be allocated. This is minimal in a file where all the records are the same length. On a file composed of varying length records a maximum record length is defined to determine the size of the length metadata associated with each record.

DGerman (talk) 01:09, 7 February 2008 (UTC)

After adding all this information in the discussion page, I decided it best to basically rewrite the article. I have saved the original article if anyone wants it. It is also available in the wiki history. Tired now. In the future I may locate and include some references. DGerman (talk) 02:15, 7 February 2008 (UTC)

Too specific

While it is true that current IBM mainframe operating systems have record-oriented file systems that do not use delimitor characters, that is not universally true. Even IBM used record delimitors on the 14xx/7010, and RCA used them on several different product lines. Shmuel (Seymour J.) Metz (talk) 19:04, 1 June 2010 (UTC)

"Advantages and costs" section needs work

The first problem is that the section doesn't clearly indicate with what a record-oriented file system is being compared.

Is it being compared to a byte-stream file system such as those offered by UN*Xes and Windows, where the lowest-level file system operations are "read N bytes from the current location and advance the current location pointer by N bytes", "write N bytes to the current location and advance the current location pointer by N bytes", "move the current location pointer to byte N", "adjust the current location pointer by N bytes, N being positive or negative", and "move the current location pointer to the current end of the file and then adjust it by N bytes, N being positive or negative" (possibly with an additional operation to set the file size to a specified number of bytes)?

Is it being compared with a block-array file system, where the lowest-level file system operations are "read from N blocks starting from block M" and "write to N blocks starting from M", "block" here referring to some fixed physical block size, such as a "block" being a single disk sector? As I remember, the usual file system APIs of RSX-11M and VMS were record-oriented, but the layer offered by the file system code was more like a block array, with QIO calls to read from or write to a file, with, at least on VMS, user-mode code being required to go through RMS, but RMS, running in a more privileged mode, doing file I/O in response to requests by making those QIO calls?

Or is it being compared to other file system types, or to more than one file system type?

If it's being compared to byte-stream file systems (as used on most desktop/notebook computers, most smartphones and tablets, and a lot of servers), then note that a byte-stream file can be structured as a sequence of records, and there are frequently libraries for OSes with byte-stream files that do so (sometimes called, for example, "ISAM packages"), so it's not clear how some of the points apply.

A record oriented file has several advantages. After a program writes a collection of data as a record the program that reads that record has the understanding of those data as a collection.

What does it mean to "[have] the understanding of those data as a collection"? And, if the program that reads that record is doing so through a library that implements a record structure on a byte-stream file system, would that program also "[have] the understanding of those data as a collection"?

Often a file will contain several related records in sequence; after the program reads the beginning of the sequence, the next sequential read returns the next collection of data (record) that the writer intended to be grouped together.

That's the definition of a sequential read. Again, how is this different from a program using a record-oriented library for a byte-stream file?

Another advantage is that the record has a length and there is usually no restriction on the bit patterns composing the data record, i.e. there is no delimiter character.

Not all files in a byte-stream file system have delimiter characters. Text files typically do, but object file, executable image files, library files, database files, and many other file types do not. Many of them have structures in the file that are, in effect, records with a record length field in the record.

There is usually a cost associated with record oriented files. For fixed length records, some records may have unused space, while for variable length records the delimiter or length field takes up space. Variable length blocks may have overhead due to delimiters or length fields.

That would also apply to record structures atop a byte-oriented file system.

In addition, there is overhead imposed by the device. On a magnetic tape overhead typically takes the form of an inter-record gap.

That's a characteristic of a magnetic tape, not of a record-oriented file system; the only way to reduce that would be to accumulate many logical records in a physical record/block on the tape.

On a direct access device with fixed length sectors, there may be unused space in the last sector of a block.

That's true only if records aren't allowed to begin in the middle of a sector. Record-oriented file systems may choose to do so, so that they don't need to do the sort of buffering that byte-stream file systems do, but a library implementing records atop a byte-stream file system might also do so in order to avoid some of the buffering overhead.

On a direct access device with variable length physical records, that overhead typically takes the form of metadata and inter-record gaps.

True, although multiple logical records might be packed into a single physical record/block, just as on tape. This may be somewhat specific to S/360 and successors (and compatibles); minicomputers tended to use direct access devices with fixed block/sector sizes, as do personal computing devices and UN*X/Windows-based servers.

A major advantage of record-oriented file systems is that they abstract files kept on paper in earlier times. A record might contain data associated with a particular, e.g., building, contact, employee, part, venue.

Again, that's just a question of which software abstracts files; again, it's quite possible to implement records atop a byte-oriented - or block-array - file system.

A second motivator for the idea of record orientation is that it is in some sense the more natural orientation for persistent storage on a non-volatile but slow physical storage device. Most physical storage devices can communicate only in units of a block. Significant portions of modern operating system kernels and associated device drivers are devoted to hiding the naturally structured and delimited (and in some sense a block is just a physical record) nature of physical storage devices.

Operating system kernels, yes; the buffer caches of UN*Xes and Windows, and the per-open-file OS data structures that maintain the aforementioned current location pointer, do hide the block structure. However, given that records don't necessariy directly correspond to blocks, some code will have to hide the blocks, to some degree, from applications reading or writing records.

Associated device drivers, not really; they generally get "read from N sectors, starting at sector M of the disk" and "write to N sectors, starting at sector M of the disk" commands, with the - byte-stream, block-array, or record-oriented - file system code, some or all of which may be running in some privileged-mode section of the OS, translating block offsets within the file to physical sector numbers on the disk. (Or logical block numbers, if the disk itself maps logical block numbers to physical sector numbers; there may be further mapping with virtualized storage, etc..) Guy Harris (talk) 10:17, 23 September 2023 (UTC)

I've always read it as comparing record-oriented file systems with byte oriented file systems; both are abstractions from the underlying hardware.

The reference to overhead is generic and the reference to tape is a sepecific example; on other devices the overhead takes other forms.

I believe that QIO in VMS is a low level interface used by higher levels of RMS, not an interface for normal applications.

It might be helpful to post a separate section for each issue. — Preceding unsigned comment added by Chatul (talk • contribs) 03:23, 26 September 2023 (UTC)

S/3x0 and z/Architecture, and the (non-UN*X) OSes running on them, are special cases, given CKD drives, as I think the record structures were designed around those drives. (Although I have the impression that there are no physical CKD drives any more, just a CKD drive abstraction implemented by firmware/software atop fixed-block drives.)

For systems with fixed-block drives, including DEC systems and the hardware atop which UN*X systems and Windows run, there's a block-array file system abstraction atop which the byte-stream or record abstraction is built. Record-oriented file systems support certain forms of structuring of the data in those blocks, while byte-stream file systems can write arbitrarily-structured binary data to those blocks - that includes structuring as fixed-length records, variable-length records, variable-length records with fixed control fields, and indexed versions of those structures.

So I see the main difference being that OSes with byte-stream file APIs provide a lower-level abstraction atop which record-oriented files can be implemented, whereas OSes with record-oriented APIs don't provide that lower-level abstraction. It's not as if the latter systems can provide facilities that the former can't also provide. I.e., it's not a question of the on-disk file system layers being different, except to the extent that the on-disk file system may provide a way to associate a record format and, for fixed-length records, a record size with a file, even though the on-disk file system provides a block-array abstraction. Instead, it's a question of what file access APIs are available; the notion of "record-oriented" vs. "byte-stream" is really above the file system when the file system is viewed as providing an abstraction of a file as an array of bytes with metadata.

An OS with byte-stream oriented APIs may, or may not, provide a library that provides a record-oriented abstraction. A system with record-oriented APIs has the advantage that programmers who want to use record-structured files don't have to get a third-party library or write their own library, but "a system with record-oriented APIs" could be a system in which the lowest-level APIs available to programmers are byte-stream APIs and that includes a record-oriented library.

Yes, on VMS direct QIO access to files from user-mode code is, as far as I know, not allowed. On RSX-11M, a program can probably issue those QIOs (RSX-11M has to run on machines that have only kernel and user mode, and, as far as I know, didn't stuff RMS into supervisor mode if running on a machine with supervisor mode; RSX-11M Plus might have supported only machines with supervisor mode and may have put RMS in superviso-rmode code). However, I have the impression they're not recommended and not documented for use on files.

RMS does, according to VMS Software's RMS reference manual, support "block I/O", which appears to provide access to the block-array layer. However, if you do block writes to a file, RMS will mark the hints it maintains for the number of records in the file and the number of user data bytes in the file as being invalid. So VMS, at least, doesn't seem to offer a "pure" record-oriented abstraction.

So I'd says that "record-oriented" vs. "byte-stream" are, if you don't have hardware/firmware that enforces a certain organization of data more complicated than "fixed-size blocks", largely differences between APIs rather than between file systems, except to the extent thata file system provides metadata that implementations of record-oriented APIs might use.

For example, an ODS-5 VFS could probably be written for some UN*X, in which case byte streams would be read from or written to files. If the UN*X in question has an "extended attributes" API, it could allow reading and writing per-file metadata, and an RMS implementation could be written for that UN*X.

Similarly, it might be possible to write a VMS XQP or ACP (pluggable file system) that supported some UN*X file system that supported extended attributes, and RMS might be able to use that file system (UFS, HFS+, APFS, ZFS, ${pick_your_linux_file_system}, etc.) mostly or completely transparently. Guy Harris (talk) 05:28, 26 September 2023 (UTC)

It seems I have some gaps in my knowledge. I believe CKD first appeared with S/360 and previous disks were sectorized. I don’t understand the reasoning that led to this decision, but I think record-oriented file system is a natural fit, but I don’t know much about filesystems on the 707x or other early IBM computers, to say nothing of non-IBM systems. I started my career together with S/360, so record-oriented systems felt natural and byte-oriented filesystems very unnatural. Trying to emulate IBM’s record-oriented system in Linux for Iron-Spring PL/I I frequently found myself wishing for CKD disks to simplify other-than-sequential access to variable length records. there’s a natural fit with blocks determined by hardware. Peter Flass (talk) 13:42, 26 September 2023 (UTC)

Yes, CKD first appeared on the S/360, and most previous disks were sectorized. On the 1301 and 1302 each cylinder had a format track that controlled block sizes. You could have a mix of block sizes on a track, but each track in the cylinder had the same mix.

And, no, the underlying hardware of contemporary DASD fo z is no longer CKD; the subsystem simulates CKD or FBA on SCSI drives with a different geometry.

While CKD simplified some things, it had an overhead cost. -- Shmuel (Seymour J.) Metz Username:Chatul (talk) 14:53, 26 September 2023 (UTC)

IBM had record oriented file systems before CKD. GE never had CKD, yet GEFRC in GECOS was record oriented. AFAIK, DEC never had CKD, yet RMS is record oriented.

Both byte oriented and record oriented are abstractions, and IBM has implemented byte oriented on top of record oriented.

Access methods and file systems are closely related, but not identical. In OS/360, there were multiple access methods for Physical Sequential (PS) files and in z/OS there were (HFS is dead) multiple file systems for byte-oriented access. In Linux there are still multiple file systems. -- Shmuel (Seymour J.) Metz Username:Chatul (talk) 14:53, 26 September 2023 (UTC)

Trying to emulate IBM’s record-oriented system in Linux for Iron-Spring PL/I I frequently found myself wishing for CKD disks to simplify other-than-sequential access to variable length records. The advantage of doing so on VMS would have been that somebody at DEC already may have wished for something to to simplify other-than-sequential access to variable length records, so you wouldn't have had to solve that problem, DEC already did it for you.

I.e., this isn't a problem of a byte-stream-oriented file system, it's a problem with the disk hardware not supporting variable-length physical records.

IBM had record oriented file systems before CKD. GE never had CKD, yet GEFRC in GECOS was record oriented. AFAIK, DEC never had CKD, yet RMS is record oriented. Yes, that's why I said "S/3x0 is different" and discussed DEC systems, where the service the hardware provides is "a disk is an array of fixed-length physical blocks"; the UNIX byte-stream APIs were originally implemented on DEC hardware, and the other hardware on which UN*Xes run include, with the exception(?) of S/3x0 and z/Architecture, secondary storage providing the "a disk is an array of fixed-length physical blocks" service. (I don't know what mechanism the drivers used and use on various UN*Xes for S/3x0 and z/Architecture, including Linux; I suspect they just write fixed-length physical records.)

Both byte oriented and record oriented are abstractions, and IBM has implemented byte oriented on top of record oriented. And libraries for UN*Xes and, presumably, Windows have implemented record-oriented on top of byte-oriented. Byte-oriented and record-oriented are characteristics of APIs, not of file systems. Some operating systems provide a byte-oriented API atop which record-oriented APIs can be implemented (UN*Xes, Windows); other operating systems provide an API that includes block-oriented and record-oriented APIs and that support byte-stream text files using the record-orinted APIs (VMS).

Access methods and file systems are closely related, but not identical. Yes. For the file systems (in the "on-disk data structures and lowest-level OS code" sense) on UN*Xes, Windows, and VMS, the file system code provides either block-oriented (VMS) or byte-stream oriented (UN*Xes, Windows) access.

For VMS, directly using file system code requires special privileges, so user-mode code has to pass through what might be considered an access method, namely RMS. One access method is "block I/O", which appears to provide direct access to the block-oriented layer.

For UN*Xes and Windows, access methods can be built atop the byte-stream API.

At least in the case of VMS, ODS-5 (and ODS-2) may store metadata such as the record format (fixed-length, variable-length, variable-length with fixed control) and record length information, for the benefit of access methods, but that's the only connection between the file system and access method.

So perhaps this page should be renamed "record-oriented access method".

In Linux there are still multiple file systems. In UN*Xes since at least SunOS 2.0, there have been mechanisms into which various various file system implementations can be plugged; not all of those implementations even work atop local storage devices. The non-specialized ones (for example, not /proc or /sys or other oddball ones) provide a byte-stream service; access methods rarely, if ever, have to care which particular file system they're using. This is likely to continue to be the case, for a variety of reasons, for the near-term future; I don't see "multiple file systems" going away soon. Guy Harris (talk) 19:53, 26 September 2023 (UTC)

I like using access methods in the title, but some might view it as IBM-centric. What other vendors used the term?

A file system is associated with an API, although the API between phisical file system and logical file system might only be accessible by the access method. -- Shmuel (Seymour J.) Metz Username:Chatul (talk) 12:56, 27 September 2023 (UTC)

The term is used at least once in the RMS manual:

The record access mode (RAC) field indicates the method of retrieving or inserting records in the file; that is, whether records are read (or written) sequentially, directly, or by record file address. Only one access method can be specified for any single record operation, but you can change the record access mode between record operations.

but (if text search in PDFs can be trusted) that appears to be the only ocurrence of that phrase, and they also refer to it as an access mode.

I can't speak for other OS vendors who provided record access methods or vendors who provided record-oriented access libraries for OSes with lower-level file access APIs such as UN*Xes.

Another possibility would be "Record-oriented file"; Record-oriented file is currently a redirect to this page. This page has relatively few references, and none that appear to establish the notion that record orientation is a characteristic of a file system rather than of a file or of an API.

The various programming interfaces for files are rather different from OS to OS:

• I have the impression that OS/360's access methods constructed channel programs, making EXCP and XDAP the lowest-level API, but, in OS/360 and its successors, were the access methods trusted not to go outside the bounds of a data set, did either of those SVCs(?) reject channel programs that went outside the bounds of the data set, or was something else done?
• In RSX-11M and VMS, QIO was the lowest-level file API, with QIOs being used for "open file", "create file", "rename file", etc..
• In Windows NT, the interface that individual file system operations offer to the generic file system layer is somewhat QIO-like (which isn't surprising), but, as far as I know, the kernel-trap-level API isn't "submit an I/O Request Packet", it's calls such as NTCreateFile and byte-stream-oriented NtReadFile() and NtWriteFile() calls, atop which the byte-stream-oritned Windows file access API runs. I'm not sure where in the privileged-mode code the byte-stream abstraction is implemented; it probably involves the privileged-mode cache manager, as is the case in UN*Xes.
• In UN*Xes, the interface that individual file system operations offer to the generic file system layer is OS-dependent, but they are generally byte-stream-oriented, and procedural; the user-mode APIs are implemented atop that. Guy Harris (talk) 21:10, 27 September 2023 (UTC)

:*I have the impression that OS/360's access methods constructed channel programs, making EXCP and XDAP the lowest-level API, but, in OS/360 and its successors, were the access methods trusted not to go outside the bounds of a data set, did either of those SVCs(?) reject channel programs that went outside the bounds of the data set, or was something else done?

Somewhere in the system - I would assume IOS rather than the access methods - "When an EXCP request is submitted for a DASD device MVS gets the seek address from the IOB, validates it against the DEB to verify it is contained within an extent of the dataset. If the seek address is not contained within a valid dataset extent the request is rejected. If the seek address is valid then MVS builds a SEEK CCW using the IOB seek address. A SET FILE MASK CCW is then chained on to the SEEK CCW. The SET FILE MASK specifies what types of CCW commands may follow....There may only be one SET FILE MASK command in a CCW sequence. This keeps the user CCW program from accessing tracks outside the dataset extents. It also enforces read-only for datasets opened for input. "

Presumably you meant http://tommysprinkle.com/mvssp/category/excp-io/excp-io-introduction/ for the reference. Guy Harris (talk) 09:07, 28 September 2023 (UTC)

Or http://tommysprinkle.com/mvssp/category/excp-io/, for the entire sequence.

So "EXCP" is short for "take this channel program and insert it after some channel commands you construct to position the device to the address I gave you, as long as that address is valid, and then limit what this program can do, and execute the resulting channel program", rather than just "run this channel program". So something in between "run this channel program as is" and "do whatever is necessary to read N blocks starting at this logical block number within this file", or the byte-stream equivalent, the latter being the sort of I/O interface provided in the other OSes I mentioned. Guy Harris (talk) 09:17, 28 September 2023 (UTC)

No, EXCP is short for EXecute Channel Program and is an access method in its own right, as is the later EXCPVR. XDAP is just a front end for EXCP. Even in OS/360 EXCP does more than validate and run the channel program. Relevant to this discussion is that for both DASD and tape it prepends a CCW that limits access. For DASD it also prepends a seek (Block MX only) or does a stand-alone seek (selector only). In MVS with newer DASD it has additional functions.

MVS does have a service called STARTIO that is closer to the bare metal, but it still does a lot more than run the channel program. -- Shmuel (Seymour J.) Metz Username:Chatul (talk) 14:07, 28 September 2023 (UTC)

References

Terminology

I've been sort of following the discussion over the name of this article. It seems to me that a "filesystem" has two parts - how the data is stored on disk, and the API offered to the users. IMO "record-oriented" vs. "byte-oriented" (or stream) is determined by the API. In OS/360, before the term "filesystem" was invented, The API is structured to read "records", of various types. You could certainly treat the data as a stream, but it would be a lot more work. Likewise, in Linux systems you can impose a record structure on top of an unstructured byte stream, but it requires effort. RO vs. BO is a data structuring convention. Peter Flass (talk) 14:50, 28 September 2023 (UTC)

Yes, there are two parts. However, it's not just "record-oriented" vs. "byte-oriented"; RMS, for example, has a "block-oriented" API. I'm not sure what EXCP would be called - "physical-record-oriented"?

"Byte-oriented" and "block-oriented" are at the same level - raw data, with the only structuring being the size of the unit of data. "Byte-oriented" APIs allow I/O at any byte offset in the file, and allow reading and writing any number of bytes; "block-oriented" APIs require I/O to occur on block boundaries in the file, with block sizes typically being a power of 2, and may require that an integral number of blocks be transferred, or may allow any number of bytes to be transferred, with data at the end of the last block being discarded on read and written as something undefined or platform-defined on write. "Physical-record-oriended" is sort of block-oriented, but for CKD devices.

At the I/O device layer, except for S/3x0 and z/Architecture, I/O tends to be block-oriented, with block-oriented APIs directly mapping to that, with any byte-oriented APIs built atop that and some buffering mechanism, and record-oriented APIs built either atop a block-oriented API or, I suspect, atop a byte-oriented API used as if it were block-oriented (use file offsets and I/O byte counts that are always a multiple of the block size). At the physical-device layer on S/3x0 and z/Architecture, I/O is, as I understand it, physical-record-oriented, with record-oriented access methods built atop that.

So the first API question is whether the lowest-level APIs offered to applications are byte-oriented/block-oriented/physical-record-oriented, allowing the raw data written to the file to be seen and modified, or are record-oriented. UN*Xes (not just Linux; the byte-stream I/O dates back to Bell Labs UNIX, and Linux implements it because it's a UN*X), Windows (not sure about DOS or 16-bit Windows), and VMS all appear not to be record-oriented in that sense, as the first two of them offer a byte-oriented API and the latter appears to allow raw block-level I/O on all files, including those normally accessed by record-oriented APIs. If EXCP I/O to files is allowed to be done by arbitrary applications on arbitrary files, arguably OS/360 also offers an API layer below the record-oriented APIs.

If the answer is "yes", there's a split between byte-oriented/block-oriented (which aren't that different, especially given that byte-oriented APIs can be used in a block-oriented fashion), where all notion of "records" must be implemented on top of that layer, whether by OS APIs or third-party libraries, and physical-record-oriented, where some record orientation is provided at the lowest layer. I.e., OS/360 and successors are different from both UN*X/Windows and VMS.

After that, there's the question of whether the OS provides general-purpose record-oriented APIs or if that requires third-party libraries. (UN*Xes and Windows may provide APIs that include calls to read specific record-oriented files - I'm a core libpcap developer, and libpcap is provided with many UN*Xes and supports reading both pcap and, to some degree, pcapng files - but that's different from, for example, C-ISAM.

(And to what extent are new applications being written using record-oriented APIs, rather than using higher-level APIs, e.g. applications built against a database? For example, a bunch of applications that ship with macOS/iOS/iPadOS or that are available for those OSes are either directly built atop an SQLite database or are built atop Core Data or other Apple APIs built atop SQLite. Bigger applications may be built atop an Oracle database or a DB2 database or an SQL Server database or....) Guy Harris (talk) 21:07, 28 September 2023 (UTC)

Your "bloc" vs "record" is "Basic" vs "Queued" on OS/360. For example, QSAM deals with individual records (logical records), and BSAM deals with blocks (physical records), leaving the programmer to do whatever blocking and deblocking he wants. Peter Flass (talk) 22:45, 28 September 2023 (UTC)

Can I open an arbitrary data set with BSAM and read its raw data? (And can I get at the block's key as well?) If not, it's not quite like block access in RMS or byte-stream access in UN*X or Windows. Guy Harris (talk) 23:04, 28 September 2023 (UTC)

Yes, you can. You can read it as format U, which will read anything and tell you how long the block was. Keys are also read, if present. I've been away a while, so I'm looking some of this stuff up. You can use NOTE to get the track and block address of the last block read or written, and POINT to position to a specific block. I used to have a lot of fun with BSAM, it starts the read or write going and then you have to WAIT or CHECK for completion, so you can have lots of things going at once. Peter Flass (talk) 03:23, 29 September 2023 (UTC)

EXCP is device-oriented; the application constructs channel command words (CCWs) appropriate for the device. Block (physical record) access normally refers to an access method at a higher level.

While MVS does not support BSAM and QSAM on FBA devices in z/OS, it supported both in other operating systems. If a block on an FBA device is not a multiple of the sector size then there is wasted space.

Multics had both record and stream I/O; the streams could either be associated with segments in the file system or with, e.g., SPOOL files for unit record equipment, actual unit record equipment.

You can use BSAM and QSAM on any PS file, regardless of device type, but you cannot use either on VSAM. Technically you can't use either with ISAM, but you can use very similar equivalents. You can read keys with BSAM, and there are programs that read the VTOC that way. -- Shmuel (Seymour J.) Metz Username:Chatul (talk) 15:11, 29 September 2023 (UTC)

OK, so it sounds as if at least some OSes in the OS/360 family require that you go through the appropriate access method on a file, unlike what I infer RMS allows (namely, doing block I/O on any file) or what UN*Xes and Windows allow.

From a quick look at GE-625/635 File and Record Control Reference Manual, that's layered atop the GECOS Input/Output Supervisor, and it sounds as if you could use IOS calls on arbitrary files.

Multics has as its lowest-level I/O API "map this file into your address space as an array of words", with its record-oriented and stream-oriented APIs built atop that. If a user has permission to access a file, nothing prevents a program from just mapping that file directly. Guy Harris (talk) 20:25, 29 September 2023 (UTC)

I think you can use BSAM on any “traditional” (non-VSAM) file. You’d be crazy to use it with ISAM files, but at base all disk files are just tracks full of blocks, in a standard (CKD) format. You can, I think, just read it block-by-block. I’d like to see others weigh in here.Peter Flass (talk) 23:10, 29 September 2023 (UTC)

Yes, with some minor exceptions, e.g., handling attention interrupts from keyboard/display devices.

The term "block-by-block" is a bit misleading, since the channel program might process multiple blocks on, e.g., DASD, tape, and the channel program might only use CCWs to control the device, e.g., rewind, display message. -- Shmuel (Seymour J.) Metz Username:Chatul (talk) 14:46, 1 October 2023 (UTC)

In Multics, from an application perspective, file and I/O are very different animals. The File System is concerned with storing segments and mapping them into address spaces; its services are only relevant to non-removable DASD. The I/O System is concerned with sequential files, and supports attaching, e.g., files. segments, printers, SPOOL files, tape drives. The common I/O services used by both are at a lower level, and I don't know which rings are able to use them. -- Shmuel (Seymour J.) Metz Username:Chatul (talk) 14:46, 1 October 2023 (UTC)

It's been ages, but, as best I remember:

There were two I/O systems.

• The first was, I think, written by MIT; I forget whether calls to it were of the form iom_$<routine name> or ios_$<routine name>.
• The second was written later, by Honeywell, as a replacement for the first, and calls to it were of the form iox_$<routine name>. I seem to remember some MIT folk not thinking all that highly of it, perhaps finding it to suffer from "second system syndrome".

Both, I think, used pluggable modules to handle various I/O sources and sinks. They handled files with a module that mapped the segment or segments corresponding to a file into the process's address space, reading by copying from the segment (or, if it had a "locate mode", returning pointers to the data to be read), and writing by copying to the segment (or, if it had a "locate mode", returning a pointer to the data to be overwritten"). (Multi-segment files were, as I remember, implemented as a directory containing all the segments of the file; that was hidden below the I/O system layer. That might only have been in the new I/O system; multi-segment files were a later addition when complaints that 2^{{{1}}}18 36-bit words wasn't big enough for some files became significant.) So, for I/O to files, the I/O system used the services of the file system, not the services that the file system used.

See, for example, section 5 of Volume 2 of the Multics Programmer Manual. (As I remember, "DIM" stood for "Device Interface Module" and "IOSIM" stood for "I/O System Interface Module"; the "old I/O system" may refer to the MIT-written one.)

The new I/O system, at least, handles sequential, indexed sequential, and direct I/O, so it apperas to contain what in OS/360-and-successors-land would be called "access methods". It also handles stream I/O, with, apparently, support for seeking in files.

I think the calling I/O systems didn't involve a ring crossing, so the code runs in whatever ring the caller runs in. For doing device I/O, there's presumably a ring crossing somewhere done by the module performing the I/O; device independence might be implemented inside the I/O system, with different modules for different devices, so the ring crossing to do physical I/O may involve calls to device-specific routines, without device independence necessarily being provided by the inner-ring routines. Guy Harris (talk) 18:49, 1 October 2023 (UTC)

Notes