2014/08/19

Unsorted

Find out in UBI system, where doing the write, when data is not FF, it aborts?

http://free-electrons.com/doc/kernel-porting.pdf
get_irqnr_and_base

00691: MACHINE_START(MINI2440, "MINI2440")
00692:  /* Maintainer: Michel Pollet <buserror@gmail.com> */
00693:  .atag_offset = 0x100,
00694:  .map_io  = mini2440_map_io,
00695:  .init_machine = mini2440_init,
00696:  .init_irq = s3c2440_init_irq,
00697:  .init_time = mini2440_init_time,
00698:  .restart = s3c244x_restart,
00699: MACHINE_END

set_handle_irq(s3c24xx_handle_irq);
s3c24xx_handle_irq
  s3c24xx_handle_intc
    irq_find_mapping // getting the irq number
    handle_IRQ(irq, regs);

request_irq



PCI interrupt
http://wiki.osdev.org/Interrupts
http://en.wikipedia.org/wiki/Message_Signaled_Interrupts
http://people.na.infn.it/~garufi/didattica/CorsoAcq/PCI.Local.Bus.Specification.Revision.3.0.pdf
http://www.ics.uci.edu/~harris/ics216/pci/PCI_22.pdf
http://lwn.net/Articles/44139/

Per-CPU variable
http://stackoverflow.com/questions/16978959/how-is-percpu-pointer-implemented-in-kernel

queue_rq, called in __blk_mq_run_hw_queue, blk_mq_make_request

Null Block
default is soft_irq
null_init
  create completion_queue for each CPU.
  initialize the list of the cq.
  null_add_dev

NVMe implementation is using the single queue block layer.

NVMe doorbell
Doorbell[edit]
In a push button analogy applied to computer systems, the term doorbell or doorbell interrupt is often used to describe a mechanism whereby a software system can signal or notify a computer hardware device that there is some work to be done. Typically, the software system will place data in some well known and mutually agreed upon memory location(s), and "ring the doorbell" by writing to a different memory location. This different memory location is often called the doorbell region, and there may even be multiple doorbells serving different purposes in this region. It is this act of writing to the doorbell region of memory that "rings the bell" and notifies the hardware device that the data are ready and waiting. The hardware device would now know that the data are valid and can be acted upon. It would typically write the data to a hard disk drive, or send them over a network, or encrypt them, etc




Kernel Wait Queue
http://linuxinme.blogspot.jp/2007/06/wait-queues.html

How to choose a scheduler for block device
http://lxr.free-electrons.com/ident?i=elevator_init

mg_disk.c

blk_init_queue(mg_request_poll, ...);

mg_request
  mg_issue_req
    mg_out( ... &mg_read_intr)
    mg_do_intr = mg_read_intr


mg_probe
  platform_get_irq
  request_irq(..., mg_irq, ...)

mg_irq
  mg_do_intr //mg_read_intr
    mg_end_request


http://www.ibm.com/developerworks/library/l-timers-list/

work queue

mtd_blkdevs.c is a good example. for use of the delayed queue,

NVMe driver
driver/block/Nvme-core.c
http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2013/20130812_PreConfD_Busch.pdf

Multi Queue Request source code
block/blk-mq.c


PCI
A PCI architecture has no central DMA controller, unlike ISA. Instead, any PCI component can request control of the bus ("become the bus master") and request to read from and write to system memory. More precisely, a PCI component requests bus ownership from the PCI bus controller (usually the southbridge in a modern PC design), which will arbitrate if several devices request bus ownership simultaneously, since there can only be one bus master at one time. When the component is granted ownership, it will issue normal read and write commands on the PCI bus, which will be claimed by the bus controller and will be forwarded to the memory controller using a scheme which is specific to every chipset.

http://xillybus.com/tutorials/pci-express-tlp-pcie-primer-tutorial-guide-1
http://xillybus.com/tutorials/pci-express-tlp-pcie-primer-tutorial-guide-2

http://en.wikipedia.org/wiki/Root_complex



inode and superblock

The superblock is a data structure containing information about the filesystem as a whole. Sometimes the collective data is referred to as filesystem metadata.

This information is sometimes called file metadata (that is, data about the file’s data) and is stored in a separate data structure from the file, called the inode.

The superblock object, which represents a specific mounted filesystem.
The inode object, which represents a specific file.
The dentry object, which represents a directory entry, which is a single component of a path.
The file object, which represents an open file as associated with a process.

irq number and resource registration ...

static struct resource mvsdio_resources[] = {
[0] = {
.start = INTER_REGS_PHYS_BASE + MV_SDMMC_REGS_OFFSET,
.end = INTER_REGS_PHYS_BASE + MV_SDMMC_REGS_OFFSET + SZ_1K -1,
.flags = IORESOURCE_MEM,
},
[1] = {
.start = IRQ_GLOBAL_SDIO,
.end = IRQ_GLOBAL_SDIO,
.flags = IORESOURCE_IRQ,
},
};

#define IRQ_GLOBAL_SDIO 57




2014/08/06

Mutual Exclusion in Linux Kernel

Three kinds of Mutual Exclusion:

Mutual exclusion among different process contexts
Semaphore: down, up
Mutex: mutex_lock, mutex_unlock
Preempt disable: preempt_disable, preempt_enable

Mutual exclusion among different interrupt contexts
Hardware interrupt disable: local_irq_disable, local_irq_enable
Software interrupt disable: local_bh_disable, local_bh_enable

Mutual exclusion among different CPUs
Spin lock: spin_lock, spin_unlock
Read write spin lock: write_lock_irq, write_unlock_irq
Atomic operation: atomic_read, atomic_write
Sequence Lock: write_seqlock, write_sequnlock
RCU( Read Copy Update ): rcu_read_lock, rcu_read_unlock
Memory barrier: mb, wmb

2014/06/08

iSCSI Configuration

What is iSCSI 

iSCSI ( Internet Small Computer System Interface ) is an Internet Protocol (IP)-based storage networking standard for linking data storage facilities.

By carrying SCSI commands over IP networks, iSCSI is used to facilitate data transfers over intranets and to manage storage over long distances. iSCSI can be used to transmit data over local area networks (LANs), wide area networks (WANs), or the Internet and can enable location-independent data storage and retrieval.

The protocol allows clients (called initiators) to send SCSI commands (CDBs) to SCSI storage devices (targets) on remote servers. It is a storage area network (SAN) protocol, allowing organizations to consolidate storage into data center storage arrays while providing hosts (such as database and web servers) with the illusion of locally attached disks.

Configure Targets

In Ubuntu 14.04, simple setup: set the target device as a image file, no authentication required, can be connected from IP.
[1]Create a new image file for storage area.

# create an image file of size 20G 

root@dlp:~#
mkdir /storage
dd if=/dev/zero of=/storage/lun1.img bs=1024k count=20000

[2]Configure iSCSI Target
root@dlp:~#
aptitude -y install iscsitarget iscsitarget-dkms
root@dlp:~#
vi /etc/default/iscsitarget
# change

ISCSITARGET_ENABLE=true
root@dlp:~#
vi /etc/iet/ietd.conf
# add at the last line

# naming rule : [ iqn.yaer-month.domain:any name ]

Target iqn.2014-04.world.server:target0
 
# provided devicce as a iSCSI target

    Lun 0 Path=/storage/lun1.img,Type=fileio
 
# Alias

    Alias LUN1
 
# authentication info ( set anyone you like for "username", "password" )

    incominguser username password
root@dlp:~#
/etc/init.d/iscsitarget restart 

 * Removing iSCSI enterprise target devices:          [ OK ]
 * Starting iSCSI enterprise target service           [ OK ]
                                                      [ OK ]
root@dlp:~#
ietadm --op show --tid=1 
# show status

Wthreads=8
Type=0
QueuedCommands=32
NOPInterval=0
NOPTimeout=0


Configure Initiator

Simple Setup: No authentication, assume target IP is 192.168.11.4
[1]Configure iSCSI Initiator
root@www:~#
aptitude -y install open-iscsi
root@www:~#
vi /etc/iscsi/iscsid.conf
# discover target

root@www:~#
iscsiadm -m discovery -t sendtargets -p 192.168.11.4

192.168.11.4:3260,1 iqn.2012-07.world.server:target0
# confirm status after discovery

root@www:~#
iscsiadm -m node -o show 

# BEGIN RECORD 2.0-871 node.name = iqn.2012-07.world.server:target0
node.tpgt = 1
node.startup = manual
iface.hwaddress = <empty>
iface.ipaddress = <empty>
iface.iscsi_ifacename = default
iface.net_ifacename = <empty>
iface.transport_name = tcp
iface.initiatorname = <empty>
...
# END RECORD
# login to target

root@www:~#
iscsiadm -m node --login
...
# confirm session
root@www:~# 
iscsiadm -m session -o show 
tcp: [1] 192.168.11.4:3260,1 iqn.2012-07.world.server:target0
# confirm partitions

root@www:~#
cat /proc/partitions 




Reference

http://www.server-world.info/en/note?os=Ubuntu_12.04&p=iscsi
http://www.howtoforge.com/using-iscsi-on-ubuntu-10.04-initiator-and-target
http://en.wikipedia.org/wiki/ISCSI

2014/06/05

Flash Memory Controller

Example code:
http://lxr.linux.no/linux+v2.6.32/drivers/mtd/nand/s3c2410.c

mtd nand driver architecture:
http://kernel.org/doc/htmldocs/mtdnand.html

about ECC of nand:
http://en.wikipedia.org/wiki/Flash_memory
http://en.wikipedia.org/wiki/Error_correcting_code

bad block management
NAND devices also require bad block management by the device driver software, or by a separate controller chip. SD cards, for example, include controller circuitry to perform bad block management and wear leveling. When a logical block is accessed by high-level software, it is mapped to a physical block by the device driver or controller. A number of blocks on the flash chip may be set aside for storing mapping tables to deal with bad blocks, or the system may simply check each block at power-up to create a bad block map in RAM. The overall memory capacity gradually shrinks as more blocks are marked as bad.

nand flash controller diagram

specification of nand interface, including command definition
http://www.micron.com/~/media/Documents/Products/ONFI/onfi_31_spec.pdf

Wear Leveling
JFFS2, YAFFS, and UBIFS include bad block management, wear leveling, error correction and provide reliable filesystems for industrial use on top of NAND Flash.

Some code about Linux file system on Flash
struct nand_chip {
 int (*write_page)(struct mtd_info *mtd, struct nand_chip *chip,
   uint32_t offset, int data_len, const uint8_t *buf,
   int oob_required, int page, int cached, int raw);
            
               
}

nand_write
  nand_do_write_ops
    chip->write_page ( nand_write_page )
      ecc->write_page ( nand_write_page_hwecc )
        chip->write_buf ( s3c2410_nand_write_buf ) 
      chip->cmdfunc( nand_command ) 
        chip->cmd_ctrl( s3c2410_nand_hwcontrol )
    


    
nand_read
  nand_do_read_ops
    chip->cmdfunc ( nand_command )
      chip->cmd_ctrl( s3c2410_nand_hwcontrol )
    ecc->read_page_raw ( nand_read_page_raw )
      chip->read_buf ( s3c2410_nand_read_buf )
      

mtd_read
  mtd->_read ( nand_read )
mtd_write       
  mtd->_write ( nand_write ) 
    
    
ubi_io_write    
  mtd_write
ubi_io_read
  mtd_read  
  
Wear-Leveling is in the drivers\mtd\ubi\wl.c 


UBIFS to UBI
  ubifs_leb_read
    ubi_read
  ubifs_leb_write
    ubi_leb_write
    

UBIFS and UBI are different
  UBI source code 
    drivers\mtd\ubi
  UBIFS source code
    fs\ubifs
  UBIFS's VFS mount point
    fs\ubifs\file.c
    fs\ubifs\dir.c
    fs\ubifs\super.c

2014/01/07

Data Into Storage

I uploaded a picture to Facebook, here is a my guess how a packet of the picture flies and lands.


  • Packet is sent to Linux kernel
  • Linux kernel handles it to network adapter driver
  • Network adaptor sends packet to network, Packet goes through few routers or any other devices and  reaches server in Facebook data center.
  • Server network adapter collects data and sends it HTTP server application.
  • HTTP server application wants save this packet permanently, so the packet is sent back to server kernel to be written on a storage device.
  • Packet is sent to Fibre Channel driver, Fibre Channel adapter use Fibre Channel protocol to wrappers SCSI write command which contains the packet. 
  • Packet in SCSI command flies over SAN and arrives at a storage controller system
  • Storage controller system ( may be a RAID controller ) finds out which hard disk to write the data and wrap the data with ATA command.
  • ATA command is sent via SATA cable and received by hard disk controller
  • Hard disk controller decodes SATA information and the encode packet with EDC+LPDC and write to hard disk. ( a lot of complexity is hidden in this step, that is the my current job ...  )
There is a lot of overhead and complexity here and there. 


Reference

AHCI

AHCI Implementation
drivers/ata/ahci_platform.c
drivers/ata/libahci.c
implements most ata_port_operations operations

ahci_init_one,
  memory mapping
  hpriv->mmio = pcim_iomap_table(pdev)[ahci_pci_bar];  AHCI_PCI_BAR_STANDARD

static inline void __iomem *__ahci_port_base(struct ata_host *host,
     unsigned int port_no)
{
struct ahci_host_priv *hpriv = host->private_data;
void __iomem *mmio = hpriv->mmio;

return mmio + 0x100 + (port_no * 0x80);
}

hpriv->mmio points to the HBA memory registers,  pointed by ABAR(AHCI Base Address).
 __ahci_port_base gets the port register.


















Command table is initialized in ahci_port_start(), pp->cmd_tbl = mem;

Below maps to AHCI spec v3.3, port registers.
PORT_LST_ADDR = 0x00, /* command list DMA addr */
PORT_LST_ADDR_HI = 0x04, /* command list DMA addr hi */
PORT_FIS_ADDR = 0x08, /* FIS rx buf addr */
PORT_FIS_ADDR_HI = 0x0c, /* FIS rx buf addr hi */
PORT_IRQ_STAT = 0x10, /* interrupt status */
PORT_IRQ_MASK = 0x14, /* interrupt enable/disable mask */
PORT_CMD = 0x18, /* port command */
PORT_TFDATA = 0x20, /* taskfile data */
PORT_SIG = 0x24, /* device TF signature */
PORT_CMD_ISSUE = 0x38, /* command issue */
PORT_SCR_STAT = 0x28, /* SATA phy register: SStatus */
PORT_SCR_CTL = 0x2c, /* SATA phy register: SControl */
PORT_SCR_ERR = 0x30, /* SATA phy register: SError */
PORT_SCR_ACT = 0x34, /* SATA phy register: SActive */
PORT_SCR_NTF = 0x3c, /* SATA phy register: SNotification */
PORT_FBS = 0x40, /* FIS-based Switching */
PORT_DEVSLP = 0x44, /* device sleep */

2013/12/17

Get backtrace

This the place holder for all the investigation made about how to get back trace information at run time.

I have a embedded system running a tiny proprietary real time OS which can handle few tasks. When the system is stopped at debugger break point, from debugger it is easy to find out stack trace of the task get stopped ( or OS kernel ), but I have no way to get other task's stack trace.  ARM's debugging toolchain has little help on this since everything is proprietary.

http://www.dwarfstd.org/doc/DWARF4.pdf
https://github.com/eliben/pyelftools/wiki/User's-guide
http://eli.thegreenplace.net/2011/02/07/how-debuggers-work-part-3-debugging-information/

Post Code on Blogger

Simplest way to post code to blogger for me: <pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black;overflow-x:...