Different kinds of system components including hardware elements, software components, network connections, and sensors are required to cooperate with each other to provide various services in smart home. In recent years, we must consider how to reduce the environmental burden by energy management and to support human's life allow for a comfortable lifestyle in an aging society. This paper proposes a concept of an agent-based middleware for smart home that consists of various home electric appliances and various sensors. The agents acquire variety of information, data, etc. from the smart home environment and store/manage them in a methodical manner. In this paper, we describe the concept, design, and initial implementation based on our agent-based middleware. We implemented initial applications related to multimedia and energy management to apply for smart home.