Bayer Pattern, RAW10 Packing, and Bilinear Demosaicing

(Updated: )
Image Processing
, , ,

A Bayer image, a packed RAW stream, and an RGB image describe different stages of a camera pipeline. This article explains how a Bayer color filter array maps photosites to color samples, how MIPI RAW10 stores those samples, and how a simple bilinear algorithm reconstructs an RGB image.

Bayer CFA and sensor photosites

A Bayer pattern describes the color filter array (CFA) physically placed above the photosites of an image sensor. Each photosite measures one filtered intensity value rather than a complete RGB triplet.

The four standard 2x2 Bayer arrangements are:

1
2
3
4
RGGB    GRBG    GBRG    BGGR

R G     G R     G B     B G
G B     B G     R G     G R

The pattern repeats across the active sensor area. For example, GRBG expands to:

1
2
3
4
G R G R G R ...
B G B G B G ...
G R G R G R ...
B G B G B G ...

A standard Bayer CFA contains twice as many green samples as red or blue:

1
R : G : B = 1 : 2 : 1

Green contributes strongly to perceived luminance and spatial detail, so sampling it more densely provides a useful compromise between image quality and sensor complexity. The two sets of green samples are not separate color channels or different green filters.

For an RGGB pattern, they are sometimes identified by position:

1
2
R   Gr
Gb  B
  • Gr is green on a row that also contains red samples.
  • Gb is green on a row that also contains blue samples.

This notation is useful because red is horizontal to Gr, while blue is horizontal to Gb.

Photosites, RAW samples, and RGB pixels

Consider a 100x100 active sensor area using a 1:1 readout:

  • Every active photosite is read.
  • There is no binning, cropping, or scaling.
  • The sensor produces 100x100 Bayer samples.
  • Demosaicing produces 100x100 RGB pixels.

The spatial dimensions do not change:

flowchart TD A["100 x 100 photosites"] --> B["100 x 100 single-color Bayer samples"] B --> C["100 x 100 reconstructed RGB pixels"]

At output position (x, y), one color component is measured directly by the corresponding photosite. The other two components are estimated from neighboring samples.

A repeating 2x2 Bayer block is therefore not a super-pixel. It describes the CFA arrangement. In a 1:1 pipeline, each photosite still corresponds spatially to one output RGB pixel.

Bayer order and RAW packing are independent

The Bayer pattern answers:

Which color does the sample at (x, y) represent?

RAW packing answers:

How are sample bits arranged in memory or on the wire?

These are independent properties. A stream might be GRBG with packed RAW10, RGGB with unpacked RAW12, or another combination.

Always confirm both properties from the sensor, receiver, or file-format documentation. A correct RAW10 unpacker cannot compensate for selecting the wrong Bayer pattern, and a correct Bayer pattern cannot compensate for decoding the bytes incorrectly.

MIPI RAW10 packing

RAW10 stores one 10-bit value per photosite. In the common MIPI CSI-2 layout, four pixels occupy five bytes:

1
2
3
4
5
6
7
8
9
B0 = P0[9:2]
B1 = P1[9:2]
B2 = P2[9:2]
B3 = P3[9:2]

B4[1:0] = P0[1:0]
B4[3:2] = P1[1:0]
B4[5:4] = P2[1:0]
B4[7:6] = P3[1:0]

The samples are reconstructed as:

1
2
3
4
P0 = (B0 << 2) | ((B4 >> 0) & 0x03)
P1 = (B1 << 2) | ((B4 >> 2) & 0x03)
P2 = (B2 << 2) | ((B4 >> 4) & 0x03)
P3 = (B3 << 2) | ((B4 >> 6) & 0x03)

The following example unpacks complete groups of four pixels one row at a time:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>

bool unpack_mipi_raw10(
    const uint8_t *src,
    size_t src_stride,
    uint16_t *dst,
    size_t dst_stride,
    size_t width,
    size_t height)
{
    if (src == NULL || dst == NULL || width % 4 != 0) {
        return false;
    }

    const size_t packed_row_bytes = (width / 4) * 5;
    if (src_stride < packed_row_bytes || dst_stride < width) {
        return false;
    }

    for (size_t y = 0; y < height; ++y) {
        const uint8_t *src_row = src + y * src_stride;
        uint16_t *dst_row = dst + y * dst_stride;

        for (size_t x = 0; x < width; x += 4) {
            const uint8_t *group = src_row + (x / 4) * 5;
            const uint8_t low = group[4];

            dst_row[x + 0] =
                ((uint16_t)group[0] << 2) | ((low >> 0) & 0x03);
            dst_row[x + 1] =
                ((uint16_t)group[1] << 2) | ((low >> 2) & 0x03);
            dst_row[x + 2] =
                ((uint16_t)group[2] << 2) | ((low >> 4) & 0x03);
            dst_row[x + 3] =
                ((uint16_t)group[3] << 2) | ((low >> 6) & 0x03);
        }
    }

    return true;
}

This example deliberately requires a width divisible by four. A production decoder must follow the exact rules of its source for partial groups, row padding, line headers, and stride. Not every file described as “RAW10” is a tightly packed MIPI CSI-2 payload.

Unpacked RAW10 is commonly stored in 16-bit containers, but the byte order and bit alignment are format-specific. The 10 meaningful bits may be least-significant-bit aligned or most-significant-bit aligned.

Mapping coordinates to Bayer colors

After unpacking, each element is a value from 0 to 1023. The next step is to determine the color represented by each coordinate.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
typedef enum {
    BAYER_RGGB,
    BAYER_GRBG,
    BAYER_GBRG,
    BAYER_BGGR
} BayerPattern;

typedef enum {
    COLOR_RED,
    COLOR_GREEN,
    COLOR_BLUE
} BayerColor;

static inline BayerColor bayer_color(
    BayerPattern pattern,
    int x,
    int y)
{
    const int xe = x & 1;
    const int ye = y & 1;

    switch (pattern) {
    case BAYER_RGGB:
        return ye == 0
            ? (xe == 0 ? COLOR_RED : COLOR_GREEN)
            : (xe == 0 ? COLOR_GREEN : COLOR_BLUE);

    case BAYER_GRBG:
        return ye == 0
            ? (xe == 0 ? COLOR_GREEN : COLOR_RED)
            : (xe == 0 ? COLOR_BLUE : COLOR_GREEN);

    case BAYER_GBRG:
        return ye == 0
            ? (xe == 0 ? COLOR_GREEN : COLOR_BLUE)
            : (xe == 0 ? COLOR_RED : COLOR_GREEN);

    case BAYER_BGGR:
    default:
        return ye == 0
            ? (xe == 0 ? COLOR_BLUE : COLOR_GREEN)
            : (xe == 0 ? COLOR_GREEN : COLOR_RED);
    }
}

The RAW10 unpacking code does not change when the Bayer pattern changes. Only this coordinate-to-color interpretation and the demosaicing logic depend on the pattern.

Bilinear demosaicing for all four patterns

Bilinear demosaicing estimates missing components by averaging adjacent samples:

  • At a red position, green comes from the four axial neighbors and blue from the four diagonal neighbors.
  • At a blue position, green comes from the four axial neighbors and red from the four diagonal neighbors.
  • At a green position, red and blue come from their corresponding horizontal or vertical neighbors.

The outer border needs special handling because some neighbors fall outside the image. The following educational implementation mirrors coordinates at the edge. Mirroring by one position also preserves the Bayer parity.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
static inline int mirror_coordinate(int value, int limit)
{
    if (value < 0) {
        return -value;
    }
    if (value >= limit) {
        return 2 * limit - value - 2;
    }
    return value;
}

static inline uint16_t raw_sample(
    const uint16_t *raw,
    int width,
    int height,
    int x,
    int y)
{
    x = mirror_coordinate(x, width);
    y = mirror_coordinate(y, height);
    return raw[y * width + x];
}

static inline uint8_t raw10_to_u8(uint32_t value)
{
    return (uint8_t)((value * 255U + 511U) / 1023U);
}

bool demosaic_bilinear(
    const uint16_t *raw,
    uint8_t *rgb,
    int width,
    int height,
    BayerPattern pattern)
{
    if (raw == NULL || rgb == NULL || width < 2 || height < 2) {
        return false;
    }

    for (int y = 0; y < height; ++y) {
        for (int x = 0; x < width; ++x) {
            uint32_t red = 0;
            uint32_t green = 0;
            uint32_t blue = 0;
            const uint32_t center =
                raw_sample(raw, width, height, x, y);

            switch (bayer_color(pattern, x, y)) {
            case COLOR_RED:
                red = center;
                green = (
                    raw_sample(raw, width, height, x - 1, y) +
                    raw_sample(raw, width, height, x + 1, y) +
                    raw_sample(raw, width, height, x, y - 1) +
                    raw_sample(raw, width, height, x, y + 1)) / 4;
                blue = (
                    raw_sample(raw, width, height, x - 1, y - 1) +
                    raw_sample(raw, width, height, x + 1, y - 1) +
                    raw_sample(raw, width, height, x - 1, y + 1) +
                    raw_sample(raw, width, height, x + 1, y + 1)) / 4;
                break;

            case COLOR_BLUE:
                blue = center;
                green = (
                    raw_sample(raw, width, height, x - 1, y) +
                    raw_sample(raw, width, height, x + 1, y) +
                    raw_sample(raw, width, height, x, y - 1) +
                    raw_sample(raw, width, height, x, y + 1)) / 4;
                red = (
                    raw_sample(raw, width, height, x - 1, y - 1) +
                    raw_sample(raw, width, height, x + 1, y - 1) +
                    raw_sample(raw, width, height, x - 1, y + 1) +
                    raw_sample(raw, width, height, x + 1, y + 1)) / 4;
                break;

            case COLOR_GREEN:
                green = center;

                if (bayer_color(pattern, x ^ 1, y) == COLOR_RED) {
                    red = (
                        raw_sample(raw, width, height, x - 1, y) +
                        raw_sample(raw, width, height, x + 1, y)) / 2;
                    blue = (
                        raw_sample(raw, width, height, x, y - 1) +
                        raw_sample(raw, width, height, x, y + 1)) / 2;
                } else {
                    red = (
                        raw_sample(raw, width, height, x, y - 1) +
                        raw_sample(raw, width, height, x, y + 1)) / 2;
                    blue = (
                        raw_sample(raw, width, height, x - 1, y) +
                        raw_sample(raw, width, height, x + 1, y)) / 2;
                }
                break;
            }

            const int index = (y * width + x) * 3;
            rgb[index + 0] = raw10_to_u8(red);
            rgb[index + 1] = raw10_to_u8(green);
            rgb[index + 2] = raw10_to_u8(blue);
        }
    }

    return true;
}

This produces an interleaved RGB buffer and supports RGGB, GRBG, GBRG, and BGGR. It is intentionally simple rather than production-quality. More advanced algorithms use gradients and color differences to reduce zipper artifacts, false color, and loss of detail near edges.

Demosaicing is only one pipeline stage

Converting each 10-bit value directly to eight bits is useful for demonstrating data layout, but a real camera pipeline commonly performs additional processing:

flowchart TD A["Packed RAW"] --> B["Unpacking"] B --> C["Black-level correction"] C --> D["Defect-pixel and lens-shading correction"] D --> E["White balance"] E --> F["Demosaicing"] F --> G["Color correction"] G --> H["Tone mapping / gamma"] H --> I["RGB output"]

Without black-level correction and white balance, the output may have a strong color cast. Without gamma or tone mapping, linear sensor data often appears dark on a normal display.

The two green positions may also exhibit a small mismatch caused by the sensor or analog readout path. Some pipelines calibrate Gr and Gb separately. An algorithm may use one green sub-plane as a reference, but simply discarding the other would throw away half of the available green samples and is not a general demosaicing advantage.

Writing and identifying PNG and BMP files

A file extension does not prove the file format. Inspect the signature:

1
2
PNG: 89 50 4E 47 0D 0A 1A 0A
BMP: 42 4D

In PowerShell 7:

1
Get-Content -AsByteStream -TotalCount 16 your_image.png | Format-Hex

In Windows PowerShell 5.1:

1
Get-Content -Encoding Byte -TotalCount 16 your_image.png | Format-Hex

PNG uses lossless DEFLATE compression. A noisy image can compress poorly, so a PNG being close to BMP size does not imply that it is uncompressed.

A conventional Windows BMP contains:

1
2
3
4
14-byte BITMAPFILEHEADER
DIB header, commonly a 40-byte BITMAPINFOHEADER
optional palette or channel masks
pixel array

For an uncompressed 24-bit BMP, each row is aligned to four bytes:

1
2
3
row_size = ((width * 24 + 31) / 32) * 4
pixel_size = row_size * abs(height)
file_size = pixel_offset + pixel_size

Pixel bytes in a conventional 24-bit BMP are stored in B, G, R order. If the height is positive, rows are stored bottom-up; a negative height indicates top-down storage.

For 1920x1080, each row is already four-byte aligned:

1
1920 * 1080 * 3 = 6,220,800 bytes

That is approximately 6.22 MB, or 5.93 MiB, plus the small header.

Summary

Keep the layers of the pipeline separate:

flowchart TD A["Physical photosites and Bayer CFA"] --> B["Single-color sensor measurements"] B --> C["RAW10 packing or another storage format"] C --> D["Unpacked Bayer sample array"] D --> E["Demosaicing and color processing"] E --> F["RGB image"] F --> G["PNG, BMP, or another image file"]

The central distinctions are:

  • A photosite is a physical light-sensitive location.
  • A Bayer sample is the digital measurement from one photosite.
  • An RGB pixel is a reconstructed three-component value at that spatial position.
  • A 2x2 Bayer block is a repeating CFA pattern, not one super-pixel.
  • RAW packing controls byte layout; Bayer order controls color interpretation.