Encoding Schemes and Number System

A processor has no idea what a 5 is. Inside the chip there are only transistors, and a transistor is dependably in one of two conditions: current flowing or not flowing, high voltage or low voltage. Two states can be told apart even when the chip is hot, the power supply sags a little, and there is electrical noise on the line. Ten different voltage levels cannot be told apart that reliably. That one engineering fact is the whole reason your Aadhaar number, a UPI payment of two hundred rupees, and this sentence are all finally stored in base 2.

One two-state cell is a bit. Eight bits grouped together are a byte. With n bits you can build 2ⁿ different patterns, so a byte has 2⁸ = 256 patterns, which hold the values 0 to 255. This is not a rule someone invented; it is just counting. Each extra bit doubles the number of patterns because that bit can be 0 or 1 on top of everything the earlier bits could already do.

Positional notation. Decimal already works the way binary does, you have simply never had to think about it. In 4507 the 5 is not "five", it is "five hundred", and the only reason is where it sits. Every positional system has a base (also called the radix) and a set of legal digits running from 0 up to base minus 1. The value of the whole number is:

value = d(k) x b^k + ... + d(2) x b^2 + d(1) x b^1 + d(0) x b^0

Place numbers start at 0 at the rightmost digit and count leftwards. So (4507)₁₀ = 4x1000 + 5x100 + 0x10 + 7x1. Change the base and nothing about the method changes - only the number you raise to the power.

Base 16 runs out of ordinary digits after 9, so the letters A to F are pressed into service for the values 10 to 15. A is not a letter here, it is the number ten.

Why octal and hexadecimal exist at all. Machines are happy with binary; humans are not. Writing 2026 as 11111101010 and copying it without dropping a bit is painful. But 8 = 2³ and 16 = 2⁴, so exactly three bits collapse into one octal digit and exactly four bits collapse into one hex digit, with no arithmetic and no carrying. Decimal gets no such shortcut, because 10 is not a power of 2. That is why memory addresses, MAC addresses and web colour codes are written in hex - it is compressed binary that a human can read aloud.

Worked example. The three ideas above, checked by running them:

# A decimal number is really a sum: digit x base ** place
print(4 * 10**3 + 5 * 10**2 + 0 * 10**1 + 7 * 10**0)

# One byte = 8 bits. How many patterns does that give?
print(2**8, "patterns, values 0 to", 2**8 - 1)

# Python reads all four bases directly. All four below are the same number.
print(0b1010, 0o12, 10, 0xA)

Output:

4507
256 patterns, values 0 to 255
10 10 10 10

The last line is the point worth remembering: 0b1010, 0o12, 10 and 0xA are four ways of typing one single number. The prefix affects nothing inside the machine - once stored, all four are the same pattern of bits, and Python prints them the same way.

Second worked example. A web colour such as #FF9933 is nothing but three two-digit hex numbers stuck together - red, green, blue:

# A web colour like #FF9933 is just three hex numbers: red, green, blue
print(int("FF", 16), int("99", 16), int("33", 16))
print("largest value one pair of hex digits can hold:", int("FF", 16))

Output:

255 153 51
largest value one pair of hex digits can hold: 255

Two hex digits cover exactly 0 to 255, which is exactly one byte. That is not a coincidence - 2 hex digits are 8 bits.

Going from decimal into any other base uses two separate procedures - one for the part before the decimal point, one for the part after it. Do them separately, then join the two answers with a point.

Whole part: repeated division. Divide the number by the target base. Write down the remainder. Divide the quotient by the base again. Keep going until the quotient becomes 0. Then read the remainders bottom to top.

The reason this works is worth knowing rather than memorising. Any division gives n = quotient x b + remainder, and the remainder is always smaller than b - so the remainder is exactly the units digit of n in base b. Dividing away that remainder shifts everything one place right, and the next remainder is the next digit up. The digits therefore come out in reverse order, which is why you read them upwards.

Worked example: (173)₁₀ to binary. Each new remainder is glued to the front of the answer string, which is exactly what "read bottom-up" means:

n = 173
bits = ""
while n > 0:
    print(n, "/ 2  ->  quotient", n // 2, ", remainder", n % 2)
    bits = str(n % 2) + bits
    n = n // 2

print("Remainders read bottom-up:", bits)
print("Cross-check with Python  :", bin(173), oct(173), hex(173))

Output:

173 / 2  ->  quotient 86 , remainder 1
86 / 2  ->  quotient 43 , remainder 0
43 / 2  ->  quotient 21 , remainder 1
21 / 2  ->  quotient 10 , remainder 1
10 / 2  ->  quotient 5 , remainder 0
5 / 2  ->  quotient 2 , remainder 1
2 / 2  ->  quotient 1 , remainder 0
1 / 2  ->  quotient 0 , remainder 1
Remainders read bottom-up: 10101101
Cross-check with Python  : 0b10101101 0o255 0xad

So (173)₁₀ = (10101101)₂ = (255)₈ = (AD)₁₆. Note that (255)₈ has nothing to do with the decimal number 255 - the same three symbols mean 173 when read in base 8.

The identical method for base 8 and base 16. Only the divisor changes. For (2026)₁₀ into hexadecimal, divide by 16 each time and convert any remainder above 9 into a letter:

Reading the hex digits upwards gives 7EA, and Python confirms it: hex(2026) returns '0x7ea'. Similarly (173)₁₀ in octal is 173/8 = 21 rem 5, 21/8 = 2 rem 5, 2/8 = 0 rem 2, read upwards as 255.

Fractional part: repeated multiplication. Multiply the fraction by the base. Whatever integer part pops out on the left is the next digit. Drop that integer part and multiply the leftover fraction again. Read the digits top to bottom - the opposite direction from the whole part, because multiplying by b shifts the point one place to the right, pushing out the digit that sits closest to the point first.

Worked example: (45.375)₁₀ to binary.

frac = 0.375
for step in range(4):
    frac = frac * 2
    print("x 2  ->", frac, " carry out the integer part:", int(frac))
    frac = frac - int(frac)
    if frac == 0:
        print("remainder is 0, stop")
        break

# 45 is 101101, and .375 is .011, so 45.375 should be 101101.011
print("check integer part:", bin(45))
print("check fraction    :", 0 * 2**-1 + 1 * 2**-2 + 1 * 2**-3)

Output:

x 2  -> 0.75  carry out the integer part: 0
x 2  -> 1.5  carry out the integer part: 1
x 2  -> 1.0  carry out the integer part: 1
remainder is 0, stop
check integer part: 0b101101
check fraction    : 0.375

Digits read top-down are 0, 1, 1. So (45.375)₁₀ = (101101.011)₂.

The gotcha you must know. A decimal fraction only terminates in binary if its denominator is a power of 2. 0.375 is 3/8, so it stops after three bits. But 0.7 is 7/10, and 10 has a factor of 5, so it never stops - its bits are 1011001100110011... repeating forever. A computer must cut it off somewhere, which stores a value very slightly off. That is the honest reason 0.1 + 0.2 prints 0.30000000000000004 in Python rather than 0.3. It is not a bug in Python; it is base 2 meeting a fraction it cannot write down.

Going the other way is easier, because there is only one method and it works for every base. Number the places: 0 for the digit immediately left of the point, then 1, 2, 3 going left; and -1, -2, -3 going right from the point. Multiply each digit by base raised to its place number, then add everything.

Worked example: (110101101)₂ to decimal. Written out in full so you can see every place value. The slice b[::-1] walks the string right to left, which is the direction the place numbers grow:

# (110101101) base 2 -> decimal, by expanding place values
b = "110101101"
total = 0
place = 0
for bit in b[::-1]:
    value = int(bit) * 2**place
    total = total + value
    print("bit", bit, "at place", place, "->", bit, "x", 2**place, "=", value)
    place = place + 1

print("Total:", total)
print("Python's int() agrees:", int(b, 2))

Output:

bit 1 at place 0 -> 1 x 1 = 1
bit 0 at place 1 -> 0 x 2 = 0
bit 1 at place 2 -> 1 x 4 = 4
bit 1 at place 3 -> 1 x 8 = 8
bit 0 at place 4 -> 0 x 16 = 0
bit 1 at place 5 -> 1 x 32 = 32
bit 0 at place 6 -> 0 x 64 = 0
bit 1 at place 7 -> 1 x 128 = 128
bit 1 at place 8 -> 1 x 256 = 256
Total: 429
Python's int() agrees: 429

In an exam you do not write nine lines. Write the place values 256 128 64 32 16 8 4 2 1 above the bits, and add only the columns where the bit is 1: 256 + 128 + 32 + 8 + 4 + 1 = 429. For binary the digit is only ever 0 or 1, so there is no multiplication at all - it is pure addition of the place values you keep.

Octal and hexadecimal. Same expansion, different powers. Powers of 8 are 1, 8, 64, 512. Powers of 16 are 1, 16, 256, 4096. For hex you must first replace letters by their values: A=10, B=11, C=12, D=13, E=14, F=15.

print("(725) base 8  =", 7 * 8**2 + 2 * 8**1 + 5 * 8**0, "  int() says", int("725", 8))
print("(3AF) base 16 =", 3 * 16**2 + 10 * 16**1 + 15 * 16**0, "  int() says", int("3AF", 16))
print("(1011.101) base 2 =", int("1011", 2) + 1*2**-1 + 0*2**-2 + 1*2**-3)

Output:

(725) base 8  = 469   int() says 469
(3AF) base 16 = 943   int() says 943
(1011.101) base 2 = 11.625

So (725)₈ = 7x64 + 2x8 + 5x1 = 448 + 16 + 5 = 469, and (3AF)₁₆ = 3x256 + 10x16 + 15x1 = 768 + 160 + 15 = 943.

Digits after the point. The place values to the right of the point are b^-1, b^-2, b^-3. In binary those are 0.5, 0.25 and 0.125. So .101 means 0.5 + 0 + 0.125 = 0.625, giving (1011.101)₂ = 11 + 0.625 = 11.625. Students lose marks here by numbering the first place after the point as 0 instead of -1 - the place immediately right of the point is always -1.

Python's shortcut, and a deliberate error. int(s, base) converts a string written in any base from 2 to 36 into a decimal int. It also checks the number is legal, which is genuinely useful:

print(int("129", 8))

Output:

Traceback (most recent call last):
  File "demo.py", line 1, in 
    print(int("129", 8))
          ~~~^^^^^^^^^^
ValueError: invalid literal for int() with base 8: '129'

The error is the teaching point, not a mistake. Base 8 has no digit 9, so 129 is not an octal number at all - the same way 1G is not a hexadecimal number. Before you convert anything, check that every digit is legal for that base.

Converting binary to octal or hex through decimal is slow and invites mistakes. There is a shortcut, and it exists for a precise reason: 8 = 2³, so every possible group of 3 bits maps onto exactly one octal digit (000 to 111 covers 0 to 7 with nothing left over). And 16 = 2⁴, so every group of 4 bits maps onto exactly one hex digit (0000 to 1111 covers 0 to F). No arithmetic, no carrying - it is pure substitution.

This is also why there is no such trick for decimal. There is no whole number k with 2^k = 10, so bits never group evenly into decimal digits.

The rule. Group from the point outwards: leftwards in 3s (octal) or 4s (hex) for the whole part, padding the left with zeros; rightwards for the fractional part, padding the right with zeros. Padding on the wrong side changes the value, so this is where marks are lost.

Worked example: (101110110)₂ to octal and to hex. Nine bits split cleanly into three groups of 3, because 9 is a multiple of 3. They do not split cleanly into 4s: 9 is not a multiple of 4, so three zeros must be padded on the left to reach twelve bits, giving 0001 0111 0110.

b = "101110110"

# Octal: group the 9 bits in 3s from the right
for g in ["101", "110", "110"]:
    print(g, "->", int(g, 2))
print("octal answer:", oct(int(b, 2)))

# Hex: the same 9 bits need 3 leading zeros to make 12
for g in ["0001", "0111", "0110"]:
    print(g, "->", format(int(g, 2), "X"))
print("hex answer  :", hex(int(b, 2)))

# And backwards: each hex digit expands to exactly 4 bits
for d in "1B2":
    print(d, "->", format(int(d, 16), "04b"))
print("so (1B2) base 16 =", bin(int("1B2", 16)))

Output:

101 -> 5
110 -> 6
110 -> 6
octal answer: 0o566
0001 -> 1
0111 -> 7
0110 -> 6
hex answer  : 0x176
1 -> 0001
B -> 1011
2 -> 0010
so (1B2) base 16 = 0b110110010

So (101110110)₂ = (566)₈ = (176)₁₆, and running the process backwards, (1B2)₁₆ = (110110010)₂. Note the leading zero of 0001 is simply dropped in the final binary answer, exactly as you would drop it in decimal.

Octal to hexadecimal has no direct route. 8 and 16 are both powers of 2, but neither is a power of the other, so 3-bit groups do not line up with 4-bit groups. You must expand to binary and regroup. This is the standard three-step answer expected in the exam:

# Octal -> hexadecimal always goes through binary. (3752) base 8 is the target.
bits = ""
for d in "3752":
    bits = bits + format(int(d, 8), "03b")
print("each octal digit as 3 bits:", bits)

# Regroup the SAME 12 bits in 4s
hexdigits = ""
i = 0
while i < len(bits):
    group = bits[i:i + 4]
    print("group", group, "->", format(int(group, 2), "X"))
    hexdigits = hexdigits + format(int(group, 2), "X")
    i = i + 4

print("as hex digits:", hexdigits)
print("cross-check  :", hex(int("3752", 8)), "and decimal", int("3752", 8))

Output:

each octal digit as 3 bits: 011111101010
group 0111 -> 7
group 1110 -> E
group 1010 -> A
as hex digits: 7EA
cross-check  : 0x7ea and decimal 2026

The bits never changed - 011111101010 is the same twelve bits both times. Only the grouping changed, and that is the whole idea: 3 at a time you read it as (3752)₈, 4 at a time you read it as (7EA)₁₆, and both are the decimal number 2026. Here the twelve bits happened to divide evenly by 4, so no padding was needed; when they do not, pad on the left.

Memory holds only numbers, so text has to be turned into numbers. An encoding scheme is simply a public agreement about which number stands for which character. The agreement is the entire point - if you save a file with one scheme and open it with another, the receiver decodes different characters and you get garbage on screen.

ASCII. American Standard Code for Information Interchange. It is a 7-bit code, so it has 2⁷ = 128 codes, numbered 0 to 127. Its layout is deliberate, not random:

The gap of 32 between the cases is not an accident. 32 is 100000 in binary, so switching between upper and lower case flips one single bit - cheap for 1960s hardware to do. Extended ASCII uses the eighth bit to reach 256 codes, but there was never one agreed meaning for codes 128 to 255, and that disagreement is exactly the problem the later schemes had to solve.

Worked example: ord() and chr(). These are the two built-in functions the syllabus wants. ord() takes one character and returns its code; chr() does the reverse.

print(ord("A"), ord("Z"), ord("a"), ord("z"), ord("0"), ord("9"), ord(" "))
print(chr(65), chr(97), chr(48))

# Lowercase is exactly 32 more than uppercase, and 32 is 100000 in binary
print(ord("a") - ord("A"), bin(32))
print(chr(ord("P") + 32))

# The digit character "7" is not the number 7
print(ord("7"), ord("7") - ord("0"))

Output:

65 90 97 122 48 57 32
A a 0
32 0b100000
p
55 7

That last line matters. The character '7' has code 55; it is a picture of a digit, not the quantity seven. Subtracting ord('0') converts the picture into the number.

ISCII. Indian Script Code for Information Interchange, standardised by the Bureau of Indian Standards as IS 13194:1991. ASCII has no place for क or ম or த, so India needed its own scheme. ISCII is an 8-bit code: codes 0 to 127 are identical to ASCII, so plain English text keeps working unchanged, and the upper half, codes 128 to 255, holds Indian script characters.

Its clever idea is that a single common chart serves many Indian scripts. Devanagari, Bengali, Gurmukhi, Gujarati, Odia, Tamil, Telugu, Kannada and Malayalam all descend from Brahmi and share the same underlying phonetic structure - the same consonants and vowels in the same order. So ISCII assigns one code per sound, and a separate script-selection code tells the display which script to draw it in. The same stored byte can appear as क or as ক depending on the script currently selected.

Why Unicode superseded it. ISCII was a sound design for 8-bit machines, but it hit hard limits:

• Only the 128 codes of the upper half were left for all of India's characters.
• Because script selection is a mode rather than part of each character's identity, a document mixing Hindi and Tamil freely is awkward, and one lost mode-switch code corrupts everything after it.
• Scripts not derived from Brahmi, such as the Perso-Arabic script used for Urdu, do not fit the shared phonetic chart.
• It solved nothing outside India. A file with Hindi and Chinese and Arabic in it was still impossible, and international software had no reason to support it.

Unicode. Unicode gives every character in every script of the world its own permanent number, called a code point, written as U+ followed by hex digits. The range runs from U+0000 to U+10FFFF, which is 1,114,112 possible code points - far more than enough. ASCII was folded in unchanged, so U+0041 is still 'A' with value 65. ISCII's work was not thrown away either: Unicode's Devanagari block, U+0900 to U+097F, was based on the ISCII layout, so the Indian standard's design survives inside the global one.

for ch in ["A", "अ", "क", "ব", "ம", "₹"]:
    print(ch, "  code point", ord(ch), " written as U+%04X" % ord(ch))

Output:

A   code point 65  written as U+0041
अ   code point 2309  written as U+0905
क   code point 2325  written as U+0915
ব   code point 2476  written as U+09AC
ம   code point 2990  written as U+0BAE
₹   code point 8377  written as U+20B9

Note that क and ব now have different code points, unlike ISCII where they could share one. Each character carries its own script identity, so one file can hold Hindi, Bengali, Tamil and English at once with no mode switching. The rupee sign ₹ is U+20B9 - which is why a UPI app or an IRCTC ticket can print it at all.

Code point versus encoding. U+0915 is क's identity. It does not say how many bytes to write to disk. That is the job of a UTF (Unicode Transformation Format), and the syllabus names two:

• UTF-8 is variable-length, using 1 to 4 bytes per character. U+0000 to U+007F take 1 byte, U+0080 to U+07FF take 2, U+0800 to U+FFFF take 3, and U+10000 upwards take 4. Its great advantage is that the 1-byte range is byte-for-byte identical to ASCII, so every old English file is already valid UTF-8. Devanagari falls in the 3-byte band. UTF-8 is the default on the web.
• UTF-32 is fixed-length: every character takes exactly 4 bytes, holding the code point directly. Finding the 500th character is instant arithmetic, but plain English text becomes four times larger than it needs to be.

for ch in ["A", "क", "₹"]:
    u8 = ch.encode("utf-8")
    u32 = ch.encode("utf-32-be")
    print(ch, "| UTF-8:", len(u8), "bytes", u8.hex(" "), "| UTF-32:", len(u32), "bytes", u32.hex(" "))

msg = "Namaste नमस्ते"
print(msg, "has", len(msg), "characters")
print("UTF-8 needs ", len(msg.encode("utf-8")), "bytes")
print("UTF-32 needs", len(msg.encode("utf-32-be")), "bytes")

Output:

A | UTF-8: 1 bytes 41 | UTF-32: 4 bytes 00 00 00 41
क | UTF-8: 3 bytes e0 a4 95 | UTF-32: 4 bytes 00 00 09 15
₹ | UTF-8: 3 bytes e2 82 b9 | UTF-32: 4 bytes 00 00 20 b9
Namaste नमस्ते has 14 characters
UTF-8 needs  26 bytes
UTF-32 needs 56 bytes

Read the UTF-32 bytes for क: 00 00 09 15. That is U+0915 written out directly, which is exactly what fixed-length means. The UTF-8 bytes e0 a4 95 look nothing like 0915 because UTF-8 spreads the code point across three bytes with marker bits. And the count at the end shows the trade-off plainly: 14 characters cost 26 bytes in UTF-8 (8 English characters at 1 byte each, 6 Devanagari at 3 bytes each) but a flat 14 x 4 = 56 bytes in UTF-32.