This is a work in progress but I'm pretty sure of the basics...
The number on the side of the nose of U.S. Navy carrier-based airplanes is often referred to as the modex number (I don’t know of a USAF equivalent). In fact, strictly speaking, it is only one element of the modex number and is in fact, more accurately the “Unit Aircraft Numeral” [MIL-I-18464 (Aer)] or more colloquially, side number.
The "NH" on the vertical fin of this F3H-2N in 1960 is the Unit Identifier or tail code (for posts discussing tail codes, click HERE and HERE) and the number under it is the last four digits of the Bureau Number, also known as a call number. Note that some units that were only assigned a few of a particular type of aircraft typically used the last three digits (occasionally four in the event of a duplication of the first three) of the BuNo for a side number on the nose.
The term modex apparently didn't exist until the early 1960s, perhaps coincidental with the introduction of the Grumman E-2A with an advanced Airborne Early Warning radar system that used the octal number system for computations like civil ATC's Mode 3 use of a four-digit octal code (4096 possible combinations, no 8s or 9s). It therefore resulted in the elimination of side numbers with the digits 8 and 9 (some arguably knowledgeable experts believe that this was also the result of the introduction of an early maintenance reporting system with a similar limitation). At some point thereafter it became fashionable but incorrect to refer to an aircraft’s side number as the modex number even though it was only part of it and even refer to a side number an aircraft in a picture taken before the early 1960s as a modex.
The Modex number was actually a four-digit code used to identify a specific aircraft by means of its Mode 2 Identification Friend or Foe (IFF) capability. According to Jan Jacobs, a former F-4 and F-14 RIO, it was not resettable by the pilot like the Mode 3 code, which was assigned by civil Air Traffic Control for radar aircraft identification. The first number was peculiar to the Air Wing that the squadron was assigned to and the second, third, and fourth digits were the aircraft’s side number. For example, Jan wrote “Air Wing 2's unique code was "7" so the CAG bird from VF-21 would have a 7200 Mode 2 squawk. The skipper's bird would be 7201. The AWG-10 radar in the F-4J had the capability of reading Mode 2 or 3 squawks by using the APX-76 IFF interrogator system. It was very handy in finding your tanker overhead the ship when there were 2 or more tankers on station. The RIO would set in the tanker's (modex) and the aircraft's distance and azimuth would show up as a double-bar on the radar scope.”
Jan goes on to describe why the side numbers never included the digits 8 or 9: “The 50's/60's-era IFF systems used 8-bit data systems (Base 8). Digits available were 0 through 7 (a total of 8 bits). In this system, 8 and 9 did not exist as 1-digit numbers. Mode 2 used this system so no aircraft that were in a tactical air wing could use 8 or 9 in their (side numbers) because Mode 2 did not support it. You'll see RAG aircraft and other Navy aircraft with 8 and 9 in their side numbers, but they were not in a tactical air wing.”
Presumably the limitation of the first digit to 8 numbers (if 0 is included) was not a problem since there were never more than a few air groups flying in one geographical area.
There is one somewhat cryptic modex statement in the subsequent Paint Schemes and Exterior Markings document, MIL-STD-2161A dated 1 May 1993: “MODEX numbers applied to aircraft with the tactical paint scheme shall be FED-STD-595, color number 36081…”. There is no definition of “MODEX number” in that Standard Practice document but there might be in subsequent revisions. More relevant and not inconsistent with this statement, assuming that it only applies to the tail code as well as the side number, is this Call Sign Requirements statement in NATOPS General Flight and Operating Instructions dated 23 November 2009 (it may not be current): Call sign selection for cross-country flights shall be made in accordance with DoD FLIPS. It is strongly recommended that squadron modex (NJ213, DB214) be used in flight planning. If the use of tactical/squadron call signs is necessary, call signs shall be the approved JANAP 119 call sign for the unit concerned. Abbreviations or contractions of these call signs in not authorized.
n.b. I had thought that the typical call-sign used in civil-aviation circumstances was the word Navy or Marine as appropriate followed by the last three or four digits of the Bureau Number.
The practice by some to caption a picture taken after the mid 1960s referring to the combination of the tail code letter(s) and side number (in that order) as a modex would be legitimized by this NATOPS statement but an anachronism for one taken earlier.
I have yet to find an explanation of the origin of the term Modex, other than it may hark back to the post-war "Mark X" IFF system that could identify a specific aircraft as opposed to simply friend or foe. TACAN systems have a Mode X and Y but those denote two different frequency ranges.
Knowledgeable (e.g. not Wikipedia) comments, additions, and corrections welcome.