How video frame catched.

VSYNC  indicates when a new image is about to start being sent out on the pixel data bus.

HREF indicates when a new line is about to start being sent out on the pixel data bus.

PCLK indicates when there is valid pixel data on the pixel data bus.

The VIP engine latches data into the VIP buffers during the following conditions:
• On the active EDGE of VSYNC (0 = falling-edge, 1 = rising-edge), the VIP engine will count until the number of expected lines are received (i.e. any extraneous lines are ignored).
• On the active LEVEL of HREF (0 = active high, 1 = active low), the VIP engine will count until the number of expected pixels are received (i.e. any extraneous pixels are ignored).

FFmpeg build and run

How to build and run ffmpeg 0.8

1. download 0.8 ffmepg

2. Install libsdl libsdl-mix, using synaptic package manager

3 ./configure --enable-gpl --enable-nonfree --enable-pthreads --enable-shared --disable-yasm --enable-debug=3 --disable-stripping --prefix=/home/xgu/ffmpeg

4. make; make install;

5. cd /home/xgu/ffmpeg/bin

6. LD_LIBRARY_PATH=../lib ./ffmpeg -s 640x480 -f video4linux2 -i /dev/video0 -f mpegts -vcodec mpeg2video out1.mpeg

7. LD_LIBRARY_PATH=../lib ./ffplay  Some_Video_File

Video Audio Capture in Linux

Video
Webcam support in Linux is mainly provided by the Linux UVC Project's UVC driver.

The UVC driver implements the Video4Linux 2 (V4L2) API

The specification of V4L2 API can be found at http://v4l2spec.bytesex.org/

Example:
In ffmpeg, file libavdevice/v4l2.c
http://v4l2spec.bytesex.org/v4l2spec/capture.c

Command:
#To capture video
ffmpeg -f video4linux2 -i /dev/video0 out.mpeg


Audio
Ubuntu is using ALSA for audio input output

Command:
#To capture audio
ffmpeg -f alsa -ac 2 -i hw:0 alsaout.wav

#To get hardware info
arecord -l



Reference:
http://ffmpeg.org/ffmpeg-doc.html
http://www.ideasonboard.org/uvc/
https://help.ubuntu.com/community/Webcam
https://help.ubuntu.com/community/UVC
http://v4l2spec.bytesex.org/
http://www.alsa-project.org/

How Audio is Played on a certain Mobile Phone

The audio is composed of two parts: decoder and mixer.
* There can be mufti decoders active at same time,
* All the result of decoders goes in to mixer
* Input of decoder is encoded frames, out put of decoder is PCM
* input of mixer is connected to the output decoder, mixer mix all the streams to one stream.

aac -- aac decoder -- PCM \
amr -- amr decoder -- PCM - Mixer -- PCM
mp3 -- mp3 deocder -- PCM /

There is one DSP. aac/amr/mp3 decoder applet and mixer applet share the DSP resource.
A non-preemptive OS is controlling all the applets.

A DSP session example
AMR Decoder --> Volume Control --> FormatConversion --> Sample Rate Conversion --> Effects --> Audio Render.

Audio Video Sync Knowledge

This is how AV sync is done
Audio time is get and set to video processing module.
video block compare the current video time and audio time and decide the should play video faster or slower.

How video can play video faster or slower
Video blocks decodes a frame and then display it on the screen, by extending or shortening the display time, video can be played faster or slower. Be noticed that decoding time can never be reduced and no frame is dropped.

AV mis-sync example
1. We consume too much time in video decoding process, so that even we display the decoded video frame at shortest time, video still can not catch up with audio.
This is how we spent too much time in video decoding. We align every byte of video ring-buffer before we decode one frame, and the video is 30fps, which will consume a lot of computation time.

2. A/V mis-sync happened during encoding. This problem happens because start of audio encoding is too much late than start of video encoding, And all the first audio/video encoded frame starts with the timestamp 0. So when we play the encoded video file, we can see the video is delayed.

3. A/V mis-sync happened during VT. The network only has max 48kbps bandwidth to transfer encoded video frames. But at the start of the video encoding, the bitrate at the first second burst into 56kbps( then drop to 48kbps as we set). This will the delay the transfer of the video data for a about 0.5( or 2?) secs. As in the coming time, all the 48kbps bandwidth is still fully occupied, the delayed part can never find a chance to find time to do a make up. So the delay happend at the beginning will continue stably... turn no worse or better.