PCIE Configuration & Enumeration

Configure Space Addressing
One of the major improvements the PCI Local Bus had over other I/O architectures was its configuration mechanism. In addition to the normal memory-mapped and I/O port spaces, each device function on the bus has a configuration space, which is 256 bytes long, addressable by knowing the eight-bit PCI bus, five-bit device, and three-bit function numbers for the device (commonly referred to as the BDF or B/D/F, as abbreviated from bus/device/function)


About Device ID and Vendor ID
The Device ID (DID) and Vendor ID (VID) registers identify the device (such as an IC), and are commonly called the PCI ID. The 16-bit vendor ID is allocated by the PCI-SIG. The 16-bit device ID is then assigned by the vendor. There is an ongoing project to collect all known Vendor and Device IDs. (See the external links below.)


Software Implementation
First method Addressing a device via Bus, Device, and Function (BDF) is also referred to as "addressing a device geographically."
  See arch/x86/pci/early.c in the Linux kernel code for an example of code that uses geographical addressing.
https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/arch/x86/pci/early.c?id=refs/tags/v3.12.7

The second method was created for PCI Express. It is called Enhanced Configuration Access Mechanism (ECAM). It extends device's configuration space to 4k, with the bottom 256 bytes overlapping the original (legacy) configuration space in PCI. The section of the addressable space is "stolen" so that the accesses from the CPU don't go to memory but rather reach a given device in the PCI Express fabric. During system initialization, firmware determines the base address for this “stolen” address region and communicates it to the root complex and to the operating system. This communication method is implementation-specific, and not defined in the PCI Express specification.

Enumeration
Since there is no direct method for the BIOS or operating system to determine which PCI slots have devices installed (nor which functions the device implements) the PCI bus(es) must be enumerated. Bus enumeration is performed by attempting to read the vendor register and device ID (VID/DID) register for each combination of bus number and device number at the device's function #0. Note that device number, different from DID, is merely a device's sequential number on that bus, moreover, after a new bridge a new bus number is defined and device enumeration restart by zero.

The process by which configuration software discovers the system topology and assigns
bus numbers and system resources.
On x86 PCIe hierarchy enumeration done by BIOS on hardware initialization state – all
registers configured before bootloader.
System software can re-assign enumeration according to enumeration rules.

Reference
https://en.wikipedia.org/wiki/PCI_configuration_space

PCI MSI Setup

The Root Complex(RC) controller receives the MSI messages send from PCI device, then RC raise an interrupt to CPU.

During the startup, software tells PCI device the address to where MSI message should be written. The address is mapped to RC controller IO space.

An example implementation:
http://lxr.free-electrons.com/source/arch/arm/mach-iop13xx/msi.c

The call stack:
__write_msi_msg
write_msi_msg
arch_setup_msi_irq

in arch_setup_msi_irq(), the address (to where MSI message is written) is defined as below.
  msg.address_hi = 0x0;
  msg.address_lo = IOP13XX_MU_MIMR_PCI;

in __write_msi_msg(), the address is actually written to device configuration register (message address register).
http://lxr.free-electrons.com/source/drivers/pci/msi.c#L327

PCI BAR Assignment

Question:
How PCI BAR is assigned in Linux?

Answer:
Find free space from PCI bus resource in __find_resource, in which resource start and end are allocated ( from the bus IO memory region? )
__pci_assign_resource
pci_bus_alloc_resource
pci_bus_alloc_from_region
allocate_resource
reallocate_resource
__find_resource

In below function, parameter resno is the BAR number. 

static int __pci_assign_resource(struct pci_bus *bus, struct pci_dev *dev, int resno, resource_size_t size, resource_size_t align)

PCI BAR address points device IO. The value of the BAR address is just mapped by OS, not in range of the host physical memory. 

To update the BAR to PCI device. 

pci_assign_resource  // from the res->start ... 

  pci_update_resource

    pcibios_resource_to_bus(dev->bus, &region, res);

      region->start = res->start - offset;

    new = region.start | (res->flags & PCI_REGION_FLAG_MASK);

    reg = pci_resource_bar(dev, resno, &type);   

    pci_write_config_dword(dev, reg, new);

Reference:

http://www.linux-mips.org/wiki/PCI_Subsystem

http://wiki.osdev.org/PCI